Original: O N S ELF -S UPERVISED I MAGE R EPRESENTATIONS FOR GAN E VALUATION
Revised: O N S ELF -S UPERVISED I MAGE R EPRESENTATIONS FOR GAN E VALUATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The embeddings from CNNs pretrained on Imagenet classiﬁcation are de-facto standard image representations for assessing GANs via FID, Precision and Recall measures.
Revised: The embeddings from CNNs pretrained on Imagenet classiﬁcation are de-facto standard image representations for assessing GANs via FID, Precision and Recall measures.

Original: Despite broad previous criticism of their usage for non-Imagenet domains, these embeddings are still the top choice in most of the GAN literature.
Revised: Despite broad previous criticism of their usage for non-Imagenet domains, these embeddings are still the top choice in most of the GAN literature.

Original: In this paper, we advocate the usage of the state-of-the-art self-supervised representations to evaluate GANs on the established non-Imagenet benchmarks.
Revised: In this paper, we advocate the usage of the state-of-the-art self-supervised representations to evaluate GANs on the established non-Imagenet benchmarks.

Original: These representations, typically obtained via contrastive learning, are shown to provide better transfer to new tasks and domains, therefore, can serve as more universal embeddings of natural images.
Revised: These representations, typically obtained via contrastive learning, are shown to provide better transfer to new tasks and domains, therefore, can serve as more universal embeddings of natural images.

Original: With extensive comparison of the recent GANs on the common datasets, we show that self-supervised representations produce a more reasonable ranking of models in terms of FID/Precision/Recall, while the ranking with classiﬁcation-pretrained embeddings often can be misleading.
Revised: With extensive comparison of the recent GANs on the common datasets, we show that self-supervised representations produce a more reasonable ranking of models in terms of FID/Precision/Recall, while the ranking with classiﬁcation-pretrained embeddings often can be misleading.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial networks (GANs) are an extremely active research direction inthe machine learning.
Revised: Generative adversarial networks (GANs) are an extremely active research direction in machine learning.

Original: The intensive development of the ﬁeld requires established quantitative measures to assess constantly appearing models.
Revised: The intensive development of the ﬁeld requires established quantitative measures to assess constantly appearing models.

Original: While a large number of evaluation protocols were proposed (Borji, 2019; Xu et al., 2018; Zhou et al., 2019; Naeem et al., 2020), there is still no consensus regarding the best evaluation measure.
Revised: While a large number of evaluation protocols were proposed (Borji, 2019; Xu et al., 2018; Zhou et al., 2019; Naeem et al., 2020), there is still no consensus regarding the best evaluation measure.

Original: Across the existing measures, the Fr´echet Inception Distance (FID) (Heusel et al., 2017) and Precision/Recall (Kynk ¨ a ¨ anniemi et al., 2019) are the most widely adopted due to their simplicity and decent consistency with human judgments.
Revised: Across the existing measures, the Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) and Precision/Recall (Kynk ¨ a ¨ anniemi et al., 2019) are the most widely adopted due to their simplicity and decent consistency with human judgments.

Original: FID and Precision/Recall quantify the discrepancy between distributions of real and generated images.
Revised: FID and Precision/Recall quantify the discrepancy between distributions of real and generated images.

Original: Since these distributions are complicated to describe in the original RGB space, the images are represented by embeddings, typically extracted with CNNs pretrained on the Imagenet classiﬁcation task (Deng et al., 2009).
Revised: Since these distributions are complicated to describe in the original RGB space, the images are represented by embeddings, typically extracted with CNNs pretrained on the Imagenet classiﬁcation task (Deng et al., 2009).

Original: While FID computed with these embeddings was shown to correlate with human evaluation (Heusel et al., 2017), these observations were mostly obtained on the datasets, semantically close to Imagenet.
Revised: While FID computed with these embeddings was shown to correlate with human evaluation (Heusel et al., 2017), these observations were mostly obtained on the datasets, semantically close to Imagenet.

Original: Meanwhile, on non-Imagenet datasets, these embeddings can provide inadequate evaluation, as widely reported in the literature (Rosca et al., 2017; Barratt & Sharma, 2018;
Revised: Meanwhile, on non-Imagenet datasets, these embeddings can provide inadequate evaluation, as widely reported in the literature (Rosca et al., 2017; Barratt & Sharma, 2018;

Original: Zhou et al., 2019).
Revised: Zhou et al., 2019).

Original: In this work, we propose to employ the state-of-the-art self-supervised models (Chen et al., 2020;
Revised: In this work, we propose to employ the state-of-the-art self-supervised models (Chen et al., 2020a;

Original: He et al., 2020; Caron et al., 2020) to extract image embeddings for GAN evaluation.
Revised: He et al., 2020; Caron et al., 2020) to extract image embeddings for GAN evaluation.

Original: These models were shown to produce features that transfer better to new tasks, hence, they become a promising candidate to provide a more universal representation.
Revised: These models were shown to produce features that transfer better to new tasks, hence, they become a promising candidate to provide a more universal representation.

Original: Intuitively, classiﬁcation-pretrained embeddings by design can suppress the information, irrelevant for the Imagenet class labels, which, however, can be crucial for other domains, like human faces.
Revised: Intuitively, classiﬁcation-pretrained embeddings by design can suppress the information, irrelevant for the Imagenet class labels, which, however, can be crucial for other domains, like human faces.

Original: On  contrary, self-supervised models, mostly trained via contrastive learning, do not have such a bias, since their main goal is typically to learn invariances to common image augmentations.
Revised: On the contrary, self-supervised models, mostly trained via contrastive learning, do not have such a bias since their main goal is typically to learn invariances to common image augmentations.

Original: To justify the usage of self-supervised embeddings, we perform a thorough comparison of the recent GAN models trained on the ﬁve most common benchmark datasets.
Revised: To justify the usage of self-supervised embeddings, we perform a thorough comparison of the recent GAN models trained on the ﬁve most common benchmark datasets.

Original: We demonstrate that classiﬁcation-pretrained embeddings can lead to incorrect ranking in terms of FID, Precision, and
Revised: We demonstrate that classiﬁcation-pretrained embeddings can lead to incorrect ranking in terms of FID, Precision, and

Original: Recall, which are the most popular metrics.
Revised: Recall, which are the most popular metrics.

Original: On the other hand, self-supervised representations produce more reasonable ranking, advocating their advantage over “classiﬁcation-oriented” counterparts.
Revised: On the other hand, self-supervised representations produce more reasonable ranking, advocating their advantage over “classiﬁcation-oriented” counterparts.

Original: Since all the checkpoints needed to compute self-supervised embeddings are publicly available, they can serve as a handy instrument for GAN comparison, consistent between different papers.
Revised: Since all the checkpoints needed to compute self-supervised embeddings are publicly available, they can serve as a handy instrument for GAN comparison, consistent between different papers.

Original: Overall, the contributions of this paper are summarized as follows:
Revised: Overall, the contributions of this paper are summarized as follows:

Original: To the best of our knowledge, our work is the ﬁrst to employ self-supervised image representations obtained via contrastive learning to evaluate GANs trained on natural images.
Revised: To the best of our knowledge, our work is the ﬁrst to employ self-supervised image representations obtained via contrastive learning to evaluate GANs trained on natural images.

Original: By extensive experiments on the common non-Imagenet benchmarks, we demonstrate that the usage of self-supervised representations provides more accurate GAN comparison.
Revised: By extensive experiments on the standard non-Imagenet benchmarks, we demonstrate that the usage of self-supervised representations provides more accurate GAN comparison.

Original: We have evaluated a large number of publicly available GANs in terms of “self-supervised” FID, Precision and Recall, providing these values for further research as a reference.
Revised: We have evaluated a large number of publicly available GANs in terms of “self-supervised” FID, Precision, and Recall, providing these values for further research as a reference.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: GAN evaluation measures.
Revised: GAN evaluation measures.

Original: Over the last years, a variety of quantitative GAN evaluation methods have been developed by the community, and the development process has yet to converge, since all the measures possess speciﬁc weaknesses (Borji, 2019;
Revised: Over the last years, a variety of quantitative GAN evaluation methods have been developed by the community, and the development process has yet to converge since all the measures possess speciﬁc weaknesses (Borji, 2019;

Original: Xu et al., 2018).
Revised: Xu et al., 2018).

Original: The Inception Score (Salimans et al., 2016) was the ﬁrst widely adopted measure, but was shown to be hardly applicable for non-Imagenet domains (Barratt & Sharma, 2018).
Revised: The Inception Score (Salimans et al., 2016) was the ﬁrst widely adopted measure but was shown to be hardly applicable for non-Imagenet domains (Barratt & Sharma, 2018).

Original: The Fr´echet Inception Distance (FID) (Heusel et al., 2017) quantiﬁes dissimilarity of real and generated distributions, computing the Wasserstein distance between their gaussian approximations, and is currently the most popular scalar measure of GAN’s quality.
Revised: The Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) quantiﬁes dissimilarity of real and generated distributions, computing the Wasserstein distance between their Gaussian approximations, and is currently the most popular scalar measure of GAN’s quality.

Original: Several recent measures were proposed (Sajjadi et al., 2018; Kynk ¨ a ¨ anniemi et al., 2019; Naeem et al., 2020) that separately evaluate ﬁdelity and diversity of GAN-produced images.
Revised: Several recent measures were proposed (Sajjadi et al., 2018; Kynk¨a¨anniemi et al., 2019; Naeem et al., 2020) that separately evaluate ﬁdelity and diversity of GAN-produced images.

Original: All of them mostly use the embeddings produced by the Imagenet classiﬁcation CNN.
Revised: All of them mostly use the embeddings produced by the Imagenet classiﬁcation CNN.

Original: A recent work (Zhou et al., 2019) has introduced a human-in-the-loop measure, which is more reliable compared to automated ones, but cannot be used e.g. for monitoring the training process.
Revised: A recent work (Zhou et al., 2019) has introduced a human-in-the-loop measure, which is more reliable compared to automated ones but cannot be used, e.g., for monitoring the training process.

Original: We focus on three the most widely used measures: FID, Precision, and Recall, which are discussed brieﬂy below.
Revised: We focus on three the most widely used measures: FID, Precision, and Recall, which are discussed brieﬂy below.

Original: Fr´echet Inception Distance quantiﬁes the discrepancy between the distributions of real and generated images, denoted by p D and p G .
Revised: Fr´echet Inception Distance quantiﬁes the discrepancy between the distributions of real and generated images, denoted by p D and p G .

Original: Both p D and p G are deﬁned on the high-dimensional image space forming nontrivial manifolds, which are challenging to approximate by simple functions.
Revised: Both p D and p G are deﬁned on the high-dimensional image space forming nontrivial manifolds, which are challenging to approximate by simple functions.

Original: To be practical, FID operates in the lower-dimensional space of image embeddings.
Revised: To be practical, FID operates in the lower-dimensional space of image embeddings.

Original: Formally, the embeddings are deﬁned by a map f : R N → R d , where N and d correspond to the dimensionalities of the images and embeddings spaces respectively.
Revised: Formally, the embeddings are deﬁned by a map f : R N → R d , where N and d correspond to the dimensionalities of the images and embeddings spaces, respectively.

Original: By design, FID measures the dissimilarity between the induced distributions fp D , fp G as follows.
Revised: By design, FID measures the dissimilarity between the induced distributions fp D , fp G as follows.

Original: First, fp D and fp G are approximated by Gaussian distributions.
Revised: First, fp D and fp G are approximated by Gaussian distributions.

Original: Then the Wasserstein distance between these distributions is evaluated.
Revised: Then the Wasserstein distance between these distributions is evaluated.

Original: As was shown in (Dowson & Landau, 1982), for distributions deﬁned by the means µ D , µ G and the covariance matrices Σ D , Σ G , this quantity equals to (cid:107) µ D − µ G (cid:107) 22 + tr (Σ D + Σ G − 2(Σ D Σ G ) 12 ) .
Revised: As was shown in (Dowson & Landau, 1982), for distributions deﬁned by the means µ D , µ G and the covariance matrices Σ D , Σ G , this quantity equals to (cid:107) µ D − µ G (cid:107) 22 + tr (Σ D + Σ G − 2(Σ D Σ G ) 12 ) .

Original: Lower FID values correspond to higher similarity between p G and p D ; hence, can be used to evaluate the performance of generative models.
Revised: Lower FID values correspond to higher similarity between p G and p D ; hence, can be used to evaluate the performance of generative models.

Original: As a common practice in the FID computation, one typically uses the activations from the InceptionV3 (Szegedy et al., 2016) pretrained on Imagenet classiﬁcation.
Revised: As a common practice in the FID computation, one typically uses the activations from the InceptionV3 (Szegedy et al., 2016) pretrained on Imagenet classiﬁcation.

Original: Precision and Recall.
Revised: Precision and Recall.

Original: When assessing generative models, it is important to quantify both the visual quality of generated images and the model diversity, e.g., to diagnose mode collapsing.
Revised: When assessing generative models, it is important to quantify both the visual quality of generated images and the model diversity, e.g., to diagnose mode collapsing.

Original: However, the scalar FID values were shown (Sajjadi et al., 2018; Kynk ¨ a ¨ anniemi et al., 2019) to sacriﬁce diversity in favor of visual quality, therefore FID cannot serve as the only sufﬁcient metric.
Revised: However, the scalar FID values were shown (Sajjadi et al., 2018; Kynk¨a¨anniemi et al., 2019) to sacriﬁce diversity in favor of visual quality, therefore FID cannot serve as the only sufﬁcient metric.

Original: To this end, (Sajjadi et al., 2018) introduced Precision and Recall, which aim to measure the image realism and the model diversity respectively.
Revised: To this end, (Sajjadi et al., 2018) introduced Precision and Recall, which aim to measure the image realism and the model diversity, respectively.

Original: A recent follow-up (Kynk¨a¨anniemi et al., 2019) elaborates on these metrics and proposes a reasonable procedure to quantify both precision and recall based only on the image embeddings.
Revised: A recent follow-up (Kynk¨a¨anniemi et al., 2019) elaborates on these metrics and proposes a reasonable procedure to quantify both precision and recall based only on the image embeddings.

Original: In a nutshell, (Kynk¨a¨anniemi et al., 2019) assumes that the visual quality of a particular sample is high if its embedding is neighboring for the embeddings of the real images.
Revised: In a nutshell, (Kynk¨a¨anniemi et al., 2019) assumes that the visual quality of a particular sample is high if its embedding is neighboring for the embeddings of the real images.

Original: On the other hand, a given real image is considered covered by the model, if its embedding belongs to the neighborhood of embeddings of the generated images.
Revised: On the other hand, a given real image is considered covered by the model if its embedding belongs to the neighborhood of embeddings of the generated images.

Original: Self-supervised representations.
Revised: Self-supervised representations.

Original: Self-supervised learning is currently attracting much research attention, especially to the contrastive learning methods (Chen et al., 2020;
Revised: Self-supervised learning is currently attracting much research attention, especially to the contrastive learning methods (Chen et al., 2020a;

Original: He et al., 2020; Caron et al., 2020).
Revised: He et al., 2020; Caron et al., 2020).

Original: The common idea behind contrastive methods is to construct representations that are close for positive image pairs and distant for negative ones, where positive pairs are produced automatically via augmentations.
Revised: The common idea behind contrastive methods is to construct representations that are close for positive image pairs and distant for negative ones, where positive pairs are produced automatically via augmentations.

Original: The recent self-supervised methods were shown to provide more transferrable (He et al., 2020; Caron et al., 2020) and robust (Hendrycks et al., 2019) features, which implies their usage as more universal representations.
Revised: The recent self-supervised methods were shown to provide more transferrable (He et al., 2020; Caron et al., 2020) and robust (Hendrycks et al., 2019) features, which implies their usage as more universal representations.

Original: In this paper, we show them being a better alternative compared to established classiﬁer-produced embeddings in the context of GAN assessment.
Revised: In this paper, we show them being a better alternative compared to established classiﬁer-produced embeddings in the context of GAN assessment

Original:  GAN EVALUATION
Revised:  GAN EVALUATION

Original: Here we systematically compare the publicly available GANs to highlight the cases of misleading comparison with classiﬁcation-pretrained embeddings.
Revised: Here we systematically compare the publicly available GANs to highlight the cases of misleading comparison with classiﬁcation-pretrained embeddings.

Original: Our goal is to demonstrate that in these cases self-supervised embeddings are a better alternative, while in other cases the rankings with both types of embeddings are mostly consistent.
Revised: Our goal is to demonstrate that in these cases, self-supervised embeddings are a better alternative, while in other cases, the rankings with both types of embeddings are mostly consistent.

Original: We examine GANs trained on ﬁve popular benchmarks:
Revised: We examine GANs trained on ﬁve popular benchmarks:

Original: • CelebaHQ 1024x1024 (Karras et al., 2017) with the following GAN models: StyleGAN without truncation (Karras et al., 2019a), MSG (Karnewar & Wang, 2020) with truncation 0 .
Revised: • CelebaHQ 1024x1024 (Karras et al., 2017) with the following GAN models: StyleGAN without truncation (Karras et al., 2019a), MSG (Karnewar & Wang, 2020) with truncation 0 .

Original: 6 and without it, PGGAN (Karras et al., 2017).
Revised: 6 and without it, PGGAN (Karras et al., 2017).

Original: To compute the metrics, we use 30k real and synthetic images; • FFHQ 1024x1024 (Karras et al., 2019a) with the following GAN models: StyleGAN (Karras et al., 2019a), StyleGAN2 (Karras et al., 2019b), MSG (Karnewar & Wang, 2020) with truncation 0 .
Revised: To compute the metrics, we use 30k real and synthetic images; • FFHQ 1024x1024 (Karras et al., 2019a) with the following GAN models: StyleGAN (Karras et al., 2019a), StyleGAN2 (Karras et al., 2019b), MSG (Karnewar & Wang, 2020) with truncation 0 .

Original: 6 and without it.
Revised: 6 and without it.

Original: To compute the metrics, we use 30k real and synthetic images; • LSUN Bedroom 256x

Revised: To compute the metrics, we use 30k real and synthetic images; • LSUN Bedroom 256x

Original: (Yu et al., 2015) with the following GAN models: StyleGAN (Karras et al., 2019a), PGGAN (Karras et al., 2017),
Revised: (Yu et al., 2015) with the following GAN models: StyleGAN (Karras et al., 2019a), PGGAN (Karras et al., 2017),

Original: COCO-GAN
Revised: COCO-GAN

Original: (Lin et al., 2019), RPGAN (Voynov & Babenko, 2019), RPGAN with high diversity (RPGAN div.).
Revised: (Lin et al., 2019), RPGAN (Voynov & Babenko, 2019), RPGAN with high diversity (RPGAN div.).

Original: RPGAN generates 128x128 images, so we upscale them to 256x256.
Revised: RPGAN generates 128x128 images, so we upscale them to 256x256.

Original: To compute the metrics, we use 30k real and synthetic images; • LSUN Church 256x

Revised: To compute the metrics, we use 30k real and synthetic images; • LSUN Church 256x

Original: (Yu et al., 2015) with the models: StyleGAN2 (Karras et al., 2019b) with truncation 0 .
Revised: (Yu et al., 2015) with the models: StyleGAN2 (Karras et al., 2019b) with truncation 0 .

Original: 5 and without it, MSG (Karnewar & Wang, 2020) with truncation 0 .
Revised: 5 and without it, MSG (Karnewar & Wang, 2020) with truncation 0 .

Original: 6 and without it, PGGAN (Karras et al., 2017), SNGAN (Miyato et al., 2018).
Revised: 6 and without it, PGGAN (Karras et al., 2017), SNGAN (Miyato et al., 2018).

Original: SNGAN generates 128x128 images, so we upscale them to 256x256.
Revised: SNGAN generates 128x128 images, so we upscale them to 256x256.

Original: To compute the metrics, we use 100k real and synthetic images; • Imagenet 128x

Revised: To compute the metrics, we use 100k real and synthetic images; • Imagenet 128x

Original: (Deng et al., 2009) with the following GAN models: BigGAN (Brock et al., 2019), BigGAN-deep
Revised: (Deng et al., 2009) with the following GAN models: BigGAN (Brock et al., 2019), BigGAN-deep

Original: (Brock et al., 2019), S3GAN (Lucic et al., 2019), Your Local GAN (YLG) (Daras et al., 2020).
Revised: (Brock et al., 2019), S3GAN (Lucic et al., 2019), Your Local GAN (YLG) (Daras et al., 2020).

Original: To compute the metrics, we use 50k real and synthetic images (50 per class);.
Revised: To compute the metrics, we use 50k real and synthetic images (50 per class);.

Original: We include this dataset to demonstrate that for Imagenet, the proposed self-supervised representations provide consistent ranking with commonly used InceptionV3 embeddings.
Revised: We include this dataset to demonstrate that for Imagenet, the proposed self-supervised representations provide consistent ranking with commonly used InceptionV3 embeddings.

Original: To compute image embeddings we use the following publicly available models:
Revised: To compute image embeddings we use the following publicly available models:

Original: • InceptionV3 (Szegedy et al., 2016) pretrained on the ILSVRC-2012 task (Deng et al., 2009); • Resnet50 (He et al., 2016) pretrained on the ILSVRC-2012 task.
Revised: • InceptionV3 (Szegedy et al., 2016) pretrained on the ILSVRC-2012 task (Deng et al., 2009); • Resnet50 (He et al., 2016) pretrained on the ILSVRC-2012 task.

Original: We include this model since self-supervised models employ Resnet50, therefore, it is important to demonstrate that better GAN ranking comes from the training objective rather than the deeper architecture;
Revised: We include this model since self-supervised models employ Resnet50, therefore, it is important to demonstrate that better GAN ranking comes from the training objective rather than the deeper architecture;

Original: • Imagenet21k
Revised: • Imagenet21k

Original: (Kolesnikov et al., 2019) pretrained on the multi-label classiﬁcation task on approximately 14M images from the full Imagenet.
Revised: (Kolesnikov et al., 2019) pretrained on the multi-label classiﬁcation task on approximately 14M images from the full Imagenet.

Original: Kolesnikov et al.
Revised: Kolesnikov et al.

Original: (2019) have shown that supervised pretraining on huge datasets provides more transferrable features, therefore, Imagenet21k can also potentially provide more universal representations.
Revised: (2019) have shown that supervised pretraining on huge datasets provides more transferrable features, therefore, Imagenet21k can also potentially provide more universal representations.

Original: The model architecture is Resnet50; • SwAV (Caron et al., 2020) is the state-of-the-art self-supervised image representation model trained on ILSVRC-2012.
Revised: The model architecture is Resnet50; • SwAV (Caron et al., 2020) is the state-of-the-art self-supervised image representation model trained on ILSVRC-2012.

Original: The idea of SwAV is to simultaneously cluster the images, while enforcing consistency between cluster assignments produced for different augmentations of the same image.
Revised: The idea of SwAV is to simultaneously cluster the images while enforcing consistency between cluster assignments produced for different augmentations of the same image.

Original: The model architecture is Resnet50; • DeepClusterV2 (Caron et al., 2020) is another self-supervised model obtained by alternating between pseudo-labels generation via k-means clustering and training the network with a classi
Revised: The model architecture is Resnet50; • DeepClusterV2 (Caron et al., 2020) is another self-supervised model obtained by alternating between pseudo-labels generation via k-means clustering and training the network with a classiﬁcation loss supervised by these pseudo-labels.

Original: - ﬁcation loss supervised by these pseudo-labels.
Revised: The model architecture is Resnet50; • MoCoV2 (Chen et al., 2020b) is the state-of-the-art contrastive learning approach, which training objective enforces the closeness of representations produced for different augmentations of the same image while pushing apart the representations of unrelated images.

Original: The model architecture is Resnet50;
Revised: The model architecture is Resnet50.

Original: Two self-supervised models listed above outperform supervised pretraining on a number of transfer tasks (Caron et al., 2020), which implies that their embeddings capture more information relevant for these tasks, compared to supervised models pretrained on Imagenet.
Revised: Three self-supervised models listed above outperform supervised pretraining on a number of transfer tasks (He et al., 2020; Caron et al., 2020), which implies that their embeddings capture more information relevant for these tasks, compared to supervised models pretrained on Imagenet.

Original: Below, for  a large number of publicly available GANs, we present the values of FID, Precision, and Recall metrics computed with different embeddings.
Revised: Below, for [Table] a large number of publicly available GANs, we present the values of FID, Precision, and Recall metrics computed with different embeddings.

Original: For the cases, where the GANs ranking is inconsistent, we aim to show that the ranking obtained with the self-supervised representations is more reasonable.
Revised: For the cases where the GANs ranking is inconsistent, we aim to show that the ranking obtained with the self-supervised representations is more reasonable.

Original: 3. F R ´ ECHET I NCEPTION D ISTANCE
Revised: 3. F R ´ ECHET I NCEPTION D ISTANCE

Original: The FID values for the non-Imagenet datasets computed with different embeddings are shown in
Revised: The FID values for the non-Imagenet datasets computed with different embeddings are shown in

Original: Table 1.
Revised: Table 1.

Original: The cases of inconsistent ranking with supervised InceptionV3 and self-supervised SwAV embeddings are highlighted in color.
Revised: The cases of inconsistent ranking with supervised InceptionV3 and self-supervised SwAV embeddings are highlighted in color.

Original: The key observations are listed below:[Table] (a) On the CelebaHQ dataset, SwAV ranks StyleGAN higher, while InceptionV3/Resnet50 prefer
Revised: The key observations are listed below: (a) On the CelebaHQ dataset, SwAV ranks StyleGAN higher, while InceptionV3/Resnet50 prefer

Original: PGGAN.
Revised: PGGAN.

Original: Figure 1 shows random samples from both StyleGAN and PGGAN and clearly demonstrates the superiority of StyleGAN.
Revised: Figure 1 shows random samples from both StyleGAN and PGGAN and clearly demonstrates the superiority of StyleGAN.

Original: To investigate the reasons why SwAV produces  more adequate ranking compared to Inception/Resnet50, we perform two additional experiments.
Revised: To investigate the reasons why SwAV produces a more adequate ranking compared to Inception/Resnet50, we perform two additional experiments.

Original: (I)
Revised: (I)

Original: First, we verify that SwAV embeddings capture more information, relevant for face images.
Revised: First, we verify that SwAV embeddings capture more information relevant for face images.

Original: The Celeba dataset (Liu et al., 2018) provides labels of 40 attributes for each image, describing various person properties (gender, age, hairstyle, etc.).
Revised: The Celeba dataset (Liu et al., 2018) provides labels of 40 attributes for each image, describing various person properties (gender, age, hairstyle, etc.).

Original: For each attribute, we train 4-layer feedforward neural network  with 2048 neurons on each layer with cross-entropy loss, which learns to predict the attribute from the SwAV/Inception embedding.
Revised: For each attribute, we train 4-layer feedforward neural network [Table] with 2048 neurons on each layer with cross-entropy loss, which learns to predict the attribute from the SwAV/Inception embedding.

Original: For all attributes, the predictions from SwAV embeddings appear to be more accurate compared to InceptionV3 (several examples are given in Table 2).
Revised: For all attributes, the predictions from SwAV embeddings appear to be more accurate compared to InceptionV3 (several examples are given in Table 2).

Original: It conﬁrms the intuition that InceptionV3 representations partially suppress the information about small facial details, which, however, is critical to identify more realistic images.
Revised: It conﬁrms the intuition that InceptionV3 representations partially suppress the information about small facial details, which, however, is critical to identify more realistic images.

Original: (II)
Revised: (II)

Original: As a qualitative experiment,[Figure] [Table] we compare SwAV and supervised Resnet50 embeddings visually via a recent technique described in Rombach et al.
Revised: As a qualitative experiment, we compare SwAV and supervised Resnet50 embeddings visually via a recent technique described in Rombach et al.

Original: In a nutshell, this technique reveals the invariances learnt by the particular representation model: for a given image, it visualizes several images, having approximately the same embedding.
Revised: In a nutshell, this technique reveals the invariances learned by the particular representation model: for a given image, it visualizes several images, having approximately the same embedding.

Original: By inspecting these images, one can analyze what factors of variations are not captured in the embedding (see the details in Section A.1).
Revised: By inspecting these images, one can analyze what factors of variations are not captured in the embedding (see the details in Section A.1).

Original: Two illustrative examples of such visualization for SwAV and Resnet50 are shown in Figure 2, demonstrating that Resnet50 embeddings are more invariant to sensitive information, like gender or race, compared to SwAV.
Revised: Two illustrative examples of such visualization for SwAV and Resnet50 are shown in Figure 2, demonstrating that Resnet50 embeddings are more invariant to sensitive information, like gender or race, compared to SwAV.

Original: Such ignorance of sensitive information makes supervised embeddings less appealing to use as universal representations.
Revised: Such ignorance of sensitive information makes supervised embeddings less appealing to use as universal representations.

Original: [Figure]
Revised: One of the key ingredients of the visualization method is an autoencoder, which is expected to capture all relevant information from an image. However, we argue that autoencoder representations are not well-suited for evaluating generative models and elaborate on this in detail in Section D.

Original:  On  dataset, there are two inconsistencies  and 
Revised: (b) On the Bedroom dataset, there are two inconsistencies in InceptionV3 and SwAV ranking.

Original:  SwAV ranks StyleGAN higher than PGGAN and the  is that SwAV ranks RPGAN higher than COCO-GAN.
Revised: The ﬁrst is that SwAV ranks StyleGAN higher than PGGAN and the second is that SwAV ranks RPGAN higher than COCO-GAN.

Original: Figure 3 shows the samples from StyleGAN, PGGAN, RPGAN, and
Revised: Figure 3 shows the samples from StyleGAN, PGGAN, RPGAN, and

Original: COCO-GAN and demonstrates that the ranking according to SwAV embeddings is more adequate.
Revised: COCO-GAN and demonstrates that the ranking according to SwAV embeddings is more adequate.

Original: Namely, the quality of StyleGAN-generated images is substantially higher.
Revised: Namely, the quality of StyleGAN-generated images is substantially higher.

Original: Also, it is difﬁcult to identify a favorite among RPGAN and COCO-GAN visually, while InceptionV3 embeddings claim strong superiority of the COCO-GAN model.
Revised: Also, it is difﬁcult to identify a favorite among RPGAN and COCO-GAN visually, while InceptionV3 embeddings claim strong superiority of the COCO-GAN model.

Original: On the other hand, self-supervised embeddings consider these models as comparable, which is better aligned with human perception.
Revised: On the other hand, self-supervised embeddings consider these models as comparable, which is better aligned with human perception.

Original: (c) There is also a case of the inconsistent ranking of Church models, namely MSG and PGGAN.
Revised: c) There is also a case of the inconsistent ranking of Church models, namely MSG and PGGAN.

Original: But since the difference of the FID values for MSG and PGGAN is small for both InceptionV3 and
Revised: But since the difference of the FID values for MSG and PGGAN is small for both InceptionV3 and

Original: SwAV, we do not consider it as a strong disagreement.
Revised: SwAV, we do not consider it as a strong disagreement.

Original: [Figure]
Revised: (d)

Original: [Table]
Revised: The rankings with the supervised InceptionV3/Resnet50 embeddings are consistent with each other, while being different from the rankings with the self-supervised ones. It indicates that the model architecture does not affect GAN ranking, and it is the training objective that matters.

Original: other [Table]
Revised: e) Imagenet21k

Original: corrects some cases of misleading ranking with InceptionV3, 
Revised: corrects some cases of misleading ranking with InceptionV3, but not all of them.

Original: but not all of them. Namely, it correctly ranks StyleGAN and PGGAN on Bedroom, while being wrong on CelebaHQ.
Revised: Namely, it correctly ranks StyleGAN and PGGAN on Bedroom, while being wrong on CelebaHQ.

Original: (f) SwAV and DeepClusterV2 have minimal inconsistencies in the ranking of MSG* vs PGGAN on CelebaHQ and StyleGAN2* vs SNGAN on Church, but the differences in the absolute values of the FID metric are negligible, so we consider these embedding models as mostly consistent. 
Revised: (f) SwAV and DeepClusterV2 have minimal inconsistencies in the ranking of MSG* vs PGGAN on CelebaHQ and StyleGAN2* vs SNGAN on Church, but the differences in the absolute values of the FID metric are negligible, so we consider these embedding models as mostly consistent. (g) MoCoV2 ﬁxes some ranking mistakes with InceptionV3, but not all of them.

Revised: While it ﬁxes the ranking of StyleGAN and PGGAN on Bedroom and reduces the gap between RPGAN and COCOGAN, the ranking of PGGAN and StyleGAN on CelebaHQ is still incorrect.

Revised: Overall, the most reasonable rankings are obtained using SwAV/DeepCluster, which have signiﬁcantly higher transfer

Original:  Overall, self-supervised embeddings provide a more reasonable FID ranking across existing nonImagenet benchmarks.
Revised: [Figure] generated by StyleGAN is substantially higher. [Figure] Overall, self-supervised embeddings provide a more reasonable FID ranking across existing nonImagenet benchmarks.

Original: For completeness, we also report the FID values for the Imagenet dataset in
Revised: For completeness, we also report the FID values for the Imagenet dataset in

Original: Table 6.
Revised: Table 6.

Original: In this case, rankings with all embeddings are the same, which conﬁrms that the SwAV representations can be used for Imagenet as well, while it is not the main focus of our work.
Revised: In this case, rankings with all embeddings are the same, which conﬁrms that the SwAV representations can be used for Imagenet as well, while it is not the main focus of our work.

Original: 3. P RECISION
Revised: 3. P RECISION

Original: The values of the Precision metric are reported in Table 3.
Revised: The values of the Precision metric are reported in Table 3.

Original: The main observations are listed below: (a) As with FID, all supervised InceptionV3/Resnet50 embeddings provide the same ranking, except minor differences between MSG with truncation and StyleGAN on CelebaHQ, and StyleGAN and PGGAN on Church.
Revised: The main observations are listed below: (a) As with FID, all supervised InceptionV3/Resnet50 embeddings provide the same ranking, except minor differences between MSG with truncation and StyleGAN on CelebaHQ, and StyleGAN and PGGAN on Church.

Original: Self-supervised SwAV and DeepClusterV2 are also consistent except the negligible difference in the ranking of PGGAN and MSG on CelebaHQ and Church; (b) The most notable inconsistency between supervised and self-supervised embeddings is revealed on the LSUN-Church dataset, where InceptionV3 considers MSG to be comparable to StyleGAN2, while SwAV ranks StyleGAN2 signiﬁcantly higher.
Revised: Self-supervised SwAV and DeepClusterV2 are also consistent except the negligible difference in the ranking of PGGAN and MSG on CelebaHQ and Church; (b) The most notable inconsistency between supervised and self-supervised embeddings is revealed on the LSUN-Church dataset, where InceptionV3 considers MSG to be comparable to StyleGAN2, while SwAV ranks StyleGAN2 signiﬁcantly higher.

Original:  To analyze which ranking of two GANs is more seasonable, we perform the following.
Revised: (I) To analyze which ranking of two GANs is more reasonable, we perform the following.

Original: On the synthetic data from the ﬁrst GAN, we train a classiﬁer that aims to distinguish between real and synthetic images.
Revised: On the synthetic data from the ﬁrst GAN, we train a classiﬁer that aims to distinguish between real and synthetic images.

Original: This classiﬁer is then evaluated on the synthetic data from the second GAN.
Revised: This classiﬁer is then evaluated on the synthetic data from the second GAN.

Original: Concretely, we train a classiﬁer to detect synthetic images on real LSUN-Church and the images generated by MSG.
Revised: Concretely, we train a classiﬁer to detect synthetic images on real LSUN-Church and the images generated by MSG.

Original: Then we evaluate this model on hold-out real images and images produced by StyleGAN2.
Revised: Then we evaluate this model on hold-out real images and images produced by StyleGAN2.

Original: Intuitively, if a model was trained on high-quality synthetic samples, it will easily detect lower-quality ones.
Revised: Intuitively, if a model was trained on high-quality synthetic samples, it will easily detect lower-quality ones.

Original: On the other hand, if the model learns to detect only low-quality synthetics, it will be harder to discriminate real images from  the high-quality one.
Revised: On the other hand, if the model learns to detect only low-quality synthetics, it will be harder to discriminate real images from [Figure] [Table] the high-quality one.

Original: In this experiment, we employ a Resnet50 classiﬁer with a binary cross-entropy loss.
Revised: In this experiment [Table] we employ a Resnet50 classiﬁer with a binary cross-entropy loss.

Original: The results for the Church dataset are provided in Table 4, meaning that the StyleGAN2 images are of higher quality, therefore, the SwAV ranking is more reasonable. 
Revised: The results for the Church dataset are provided in Table 4, meaning that the StyleGAN2 images are of higher quality, therefore, the SwAV ranking is more reasonable. (II)

Revised: We also conduct a human study to determine which of the generative models gives more realistic images in terms of human perception.

Revised: For each generative model, we show ten assessors a real or randomly generated (fake) image and ask them to choose whether it is real or fake.

Original: There is also a minor inconsistency in the ranking of StyleGAN and StyleGAN2 on FFHQ, which we consider as negligible.
Revised: The error rate reﬂects the visual quality of the generative model. For both models, MSG and StyleGAN2, we demonstrate to assessorsimages and the error rate is 0 . 4% for MSG and 2 . 8% for StyleGAN2, which clearly shows the superiority of StyleGAN2.

Original: [Table]
Revised: [Table]

Original: (c) Imagenet21k ranks GANs less reliably compared to SwAV.
Revised: (c) Imagenet21k ranks GANs less reliably compared to SwAV.

Original: The most notable mistake is a ranking of COCO-GAN and PGGAN on Bedroom, where PGGAN produces more visually appealing images, see Figure 3.
Revised: The most notable mistake is a ranking of COCO-GAN and PGGAN on Bedroom, where PGGAN produces more visually appealing images, see Figure 3.

Original: Another case is comparison on CelebaHQ, where Imagenet21k ranks PGGAN higher that the more powerful MSG, see samples in Section B.
Revised: Another case is comparison on CelebaHQ, where Imagenet21k ranks PGGAN higher that the more powerful MSG, see samples in Section B.

Original: 3. R ECALL
Revised: 3. R ECALL

Original: The values of the Recall metric are shown in Table 5, and the main observations are provided below: 
Revised: The values of the Recall metric are shown in Table 5, and the main observations are provided below: [Table]

Original: (a) As in previous experiments, there are only minor inconsistencies between supervised InceptionV3 and Resnet50 models, namely, StyleGAN vs COCO-GAN on the Bedroom dataset and MSG vs PGGAN on the Church dataset.
Revised: a As in previous experiments, there are only minor inconsistencies between supervised InceptionV3 and Resnet50 models, namely, StyleGAN vs COCO-GAN on the Bedroom dataset and MSG vs PGGAN on the Church dataset.

Original: The only insigniﬁcant difference between the self-supervised methods is the ranking of StyleGAN with truncation vs SNGAN on the Church dataset, however,
Revised: The only insigniﬁcant difference between the self-supervised methods is the ranking of StyleGAN with truncation vs SNGAN on the Church dataset, however,

Original: Recall values for both models are negligible.
Revised: Recall values for both models are negligible.

Original: In terms of recall, Imagenet21k ranking always coincides with the ranking obtained by self-supervised methods, except the negligible discrepancy between MSG and PGGAN on the Church dataset;
Revised: In terms of Recall, Imagenet21k ranking always coincides with the ranking obtained by self-supervised methods, except the negligible discrepancy between MSG and PGGAN on the Church dataset;

Original: [Table]
Revised: [Table]

Original: (b) The absolute Recall values for SwAV/DeepClusterV2 are smaller compared to InceptionV3/Resnet50.
Revised: The absolute Recall values for SwAV/DeepClusterV2 are smaller compared to InceptionV3/Resnet50.

Original: We attribute this behavior to the fact that
Revised: We attribute this behavior to the fact that

Original: GANs tend to simplify images omitting the details (Bau et al., 2019), e.g. people in front of buildings, cars, fences, etc.
Revised: GANs tend to simplify images omitting the details (Bau et al., 2019), e.g., people in front of buildings, cars, fences, etc.

Original: The classiﬁer-pretrained embeddings are less sensitive to these details, since they are not crucial for correct classiﬁcation.
Revised: The classiﬁer-pretrained embeddings are less sensitive to these details since they are not crucial for correct classiﬁcation.

Original: In contrast, self-supervised embeddings are more susceptible to small details (see
Revised: In contrast, self-supervised embeddings are more susceptible to small details (see

Original: Figure 2 and Table 4), hence, more images are considered “not covered”.
Revised: Figure 2 and Table 4), hence, more images are considered “not covered”.

Original: Figure 5 in Section C shows examples of real LSUN-Church images that are “deﬁnitely covered” by StyleGAN2 from the standpoint of InceptionV3 embeddings, but are “not covered” if SwAV embeddings are used.
Revised: Figure 6 in Section C shows examples of real LSUN-Church images that are “deﬁnitely covered” by StyleGAN2 from the standpoint of InceptionV3 embeddings, but are “not covered” if SwAV embeddings are used.

Original: More formally, we say that a real image is covered by a synthetic one with the neighborhood size k , if the distance between their embeddings does not exceed the distance from the embedding of the synthetic image to its k -th nearest neighbor in the set of all synthetic embeddings.
Revised: More formally, we say that a real image is covered by a synthetic one with the neighborhood size k , if the distance between their embeddings does not exceed the distance from the embedding of the synthetic image to its k -th nearest neighbor in the set of all synthetic embeddings.

Original: The images from Figure 5 are covered by at least 10 synthetic images with neighborhood size 5 with InceptionV3 embeddings, while being not covered even by the neighborhood of size 100 for SwAV embeddings.
Revised: The images from Figure 6 are covered by at least 10 synthetic images with neighborhood size 5 with InceptionV3 embeddings, while being not covered even by the neighborhood of size 100 for SwAV embeddings.

Original: These images possess many small details, such as monuments, cars, people, branches in the foreground, and so on, that GANs usually omit to generate.
Revised: These images possess many small details, such as monuments, cars, people, branches in the foreground, and so on, that GANs usually omit to generate.

Original: [Table]
Revised: [Figure]

Original: c) Interestingly, the Precision values for SwAV are quite high, therefore, SwAV considers the stateof-the-art generative models being able to produce high-ﬁdelity images, but failing to generate diverse images with small details.
Revised: Interestingly, the Precision values for SwAV are quite high, therefore, SwAV considers the state-of-the-art generative models being able to produce high-ﬁdelity images, but failing to generate diverse images with small details

Original: (d) There are two signiﬁcant inconsistencies in rankings based on InceptionV3 and SwAV.
Revised: (d) There are two signiﬁcant inconsistencies in rankings based on InceptionV3 and SwAV.

Original: First, on CelebaHQ, SwAV prefers MSG* over PGGAN.
Revised: First, on CelebaHQ, SwAV prefers MSG* over PGGAN.

Original: Second, on LSUN-Bedroom, InceptionV3 ranks 
Revised: Second, on LSUN-Bedroom, InceptionV3 ranks PGGAN higher

Original: PGGAN higher than StyleGAN.
Revised: than StyleGAN.

Original: Since StyleGAN is a more powerful model compared to PGGAN, we also consider it as a case of more reliable ranking with SwAV, even though it is difﬁcult to conﬁrm quantitatively due to the lack of an “oracle” measure of generation diversity. 
Revised: Since StyleGAN is a more powerful model compared to PGGAN, we also consider it as a case of more reliable ranking with SwAV, even though it is difﬁcult to conﬁrm quantitatively due to the lack of an “oracle” measure of generation diversity. 3. S AMPLE EFFICIENCY OF S W AV- BASED FID

Revised: As an additional practical advantage of SwAV, we highlight that computing FID becomes more sample-efﬁcient compared to InceptionV3.

Revised: Namely, to obtain a reliable estimation of FID values, one requires much fewer samples when using SwAV embeddings.

Revised: We illustrate this effect in Figure 4, which plots FID values w.r.t.

Revised: sample size for StyleGAN2 trained on the Church dataset.

Original: C ONCLUSION
Revised: Since the FID values for SwAV and InceptionV3 have different typical scales, we normalize both curves by the corresponding FID value computed for a sample of size 100k. FID based on SwAV embeddings converges faster, i.e., using SwAV always achieves more reliable FID estimates for a ﬁxed sample size. We attribute this beneﬁt of SwAV to the fact that its representations capture more information needed to distinguish between real and fake distributions. Intuitively, the covariance matrices for real and synthetic data computed from SwAV embeddings are more dissimilar compared to InceptionV3based ones.

Revised: Quantitatively, the magnitude of the covariance term in FID tr ( C R + C S − 2 √ C R C S ) is larger for SwAV, which leads to smaller relative errors of its stochastic estimates.

Original:  In this paper, we have investigated if the state-of-the-art self-supervised models can produce more appropriate representations for GAN evaluation.
Revised: We elaborate on this issue more rigorously in Section E.  C ONCLUSION In this paper, we have investigated if the state-of-the-art self-supervised models can produce more appropriate representations for GAN evaluation.

Original: With extensive experiments, we have shown that using these representations often corrects the cases of misleading ranking obtained with classiﬁcation-pretrained embeddings.
Revised: With extensive experiments, we have shown that using these representations often corrects the cases of misleading ranking obtained with classiﬁcation-pretrained embeddings.

Original: Overall, self-supervised representations provide  more adequate GAN comparison on the four established non-Imagenet benchmarks of natural images.
Revised: Overall, self-supervised representations provide a more adequate GAN comparison on the four established non-Imagenet benchmarks of natural images.

Original: Of course, we do not claim that they should be used universally for all areas, e.g. for spectrograms or medical images.
Revised: Of course, we do not claim that they should be used universally for all areas, e.g., for spectrograms or medical images.

Original: But our work indicates that obtaining good representations needed for proper
Revised: But our work indicates that obtaining good representations needed for proper

Original: GAN evaluation does not require supervision, therefore, domain-speciﬁc self-supervised learning becomes a promising direction for further study.
Revised: GAN evaluation does not require supervision, therefore, domain-speciﬁc self-supervised learning becomes a promising direction for further study.

Original: A A PPENDIX A. T HE DETAILS OF THE VISUALIZATION TECHNIQUE IN S ECTION 3. 
Revised: A A PPENDIX A. T HE DETAILS OF THE VISUALIZATION TECHNIQUE IN S ECTION 3.

Original: The technique from Rombach et al.
Revised: The technique from Rombach et al.

Original: (2020) proposes to construct the embedding visualization in the following way.
Revised: (2020) proposes to construct the embedding visualization in the following way.

Original: Let us have an autoencoder A ( x ) , whose latent representations are denoted by z .
Revised: Let us have an autoencoder A ( x ) , whose latent representations are denoted by z .

Original: It is assumed that a latent representation z
Revised: It is assumed that a latent representation z

Original: = A ( x ) captures all the information from the image x , since the autoencoder’s goal it to fully reconstruct x from z .
Revised: = A ( x ) captures all the information from the image x , since the autoencoder’s goal is to fully reconstruct x from z .

Original: Let us denote the embedding model we want to study by E ( x ) .
Revised: Let us denote the embedding model we want to study by E ( x ) .

Original: For a particular image x , Rombach et al.
Revised: For a particular image x , Rombach et al.

Original: (2020) compute an autoencoder representation z
Revised: (2020) compute an autoencoder representation z

Original: = A ( x ) and aims to disentangle the information contained in the autoencoder representation z into the information contained in the image embedding e = E ( x ) and the information v to which the embedding model E ( x ) is invariant.
Revised: = A ( x ) and aims to disentangle the information contained in the autoencoder representation z into the information contained in the image embedding e = E ( x ) and the information v to which the embedding model E ( x ) is invariant.

Original: To this end, an invertible neural network (INN) is trained, which predicts the invariant part v = INN ( z, e ) based on the autoencoder representation and embedding.
Revised: To this end, an invertible neural network (INN) is trained, which predicts the invariant part v = INN ( z, e ) based on the autoencoder representation and embedding.

Original: Moreover, it is trained in such a way that the invariant part has a normal distribution v ∼ N (0 , 1) .
Revised: Moreover, it is trained in such a way that the invariant part has a normal distribution v ∼ N (0 , 1) .

Original: Finally, a bunch of images, having the same e but different invariant parts v are produced.
Revised: Finally, a bunch of images having the same e but different invariant parts v are produced.

Original: Namely, several samples of invariant parts (cid:98) v are produced from N (0 , 1) and the corrseponding latent representations are computed from (cid:98) v and embedding e , that is (cid:98) z
Revised: Namely, several samples of invariant parts (cid:98) v are produced from N (0 , 1) and the corresponding latent representations are computed from (cid:98) v and embedding e , that is (cid:98) z

Original: = INN.reverse ( v, e ) .
Revised: = INN.reverse ( v, e ) .

Original: Then (cid:98) z can be decoded into an image using the autoencoder.
Revised: Then (cid:98) z can be decoded into an image using the autoencoder.

Original: As a result, for a given embedding model E ( x ) and a given image x  it allows to obtain a set of images with approximately the same embeddings, but varying invariant part.
Revised: As a result, for a given embedding model E ( x ) and a given image x , it allows to obtain a set of images with approximately the same embeddings but varying invariant part.

Original: For the CelebaHQ dataset, we use ALAE autoencoder (Pidhorskyi et al., 2020) and train INN with the hyperparameters provided in (Rombach et al., 2020)
Revised: For the CelebaHQ dataset, we use ALAE autoencoder (Pidhorskyi et al., 2020) and train INN with the hyperparameters provided in (Rombach et al., 2020).

Original: B C ELEBA HQ SAMPLES
Revised: B C ELEBA HQ SAMPLES

Original: [Figure]
Revised: [Figure]

Original: C C HURCH IMAGES
Revised: C C HURCH IMAGES

Original: D T HE EMBEDDING MODELS CHECKPOINTS USED IN OUR STUDY
Revised: D A UTOENCODER EMBEDDINGS

Revised: In this section, we compare SwAV and ALAE autoencoder (Pidhorskyi et al., 2020) embeddings. ALAE was trained on Celeba (Liu et al., 2018) dataset, therefore, is expected to work correctly for the ranking of generative models on CelebaHQ.

Revised: To investigate what information is important for each of the embeddings, we build a dataset containing 30k real images from CelebaHQ and 30k synthetic images generated by PGGAN.

Revised: Then we select 1.5k real and 1.5k synthetic images as

Revised: [Figure]

Revised: queries and leave the remaining 57k pictures as a database.

Revised: For each query, we compute three nearest neighbors from the database.

Revised: Typical examples of the nearest neighbors are shown on Figure 7.

Revised: One can see that ALAE places a strong emphasis on the exact spatial arrangement, while sparsely sampled manifolds rarely include near-exact matches in terms of spatial structure (Kynk ¨ a ¨ anniemi et al., 2019).

Revised: This makes ALAE embeddings poor for GAN evaluation, for instance, via Precision and Recall metrics.

Revised: We also perform real/fake classiﬁcation of queries using 1-NN classiﬁcation on the constructed database.

Revised: The classiﬁcation accuracy is 0.729 for ALAE embeddings and 0.

Revised: for SwAV.

Revised: Overall, the distance in the space of the autoencoder’s embeddings is less informative to distinguish between the real/fake distributions.

Revised: E O N THE ESTIMATE OF FID VALUES

Revised: Let us denote C R the covariance matrix for real data, C S for synthetic data and (cid:101) C R and (cid:101) C S their estimates with a ﬁnite number of samples.

Revised: We also denote

Revised: [Equation]

Revised: In (Dowson & Landau, 1982) it was shown that d ( C R , C S ) deﬁnes a metric on the space of all covariance matrices of order n .

Revised: Our goal is to assess the relative error of the FID estimate.

Revised: Under the assumption that the means of real and synthetic data distributions are estimated quite accurately

Revised: (which is the case already for a few thousand of samples) we need to estimate only

Revised: [Equation]

Revised: Due to the metric properties

Revised: [Equation]

Revised: Then

Revised: [Equation]

Revised: Due to the symmetry the same inequality can be obtained with the opposite sign and as the result we get

Revised: [Equation]

Revised: where the numerator corresponds to the accuracy of the estimation of covariance matrices and the denominator corresponds to the distance between covariance matrices for real and synthetic data.

Revised: Thus, larger values of d ( C R , C S ) result in a smaller relative error of the FID estimation.

Revised: Experimentally, we compute the distances d ( C R , C S ) =

Revised: tr ( C R + C S − 2 √ C R C S ) based on SwAV and

Revised: InceptionV3 embeddings for StyleGAN2 trained on the Church dataset.

Revised: 1 We obtain 0 .

Revised: 083 for SwAV and 0 .

Revised: for InceptionV3, which conﬁrms the validity of calculations above and explains the better sample-efﬁciency of SwAV presented in Figure 4.

Revised: F

Revised: T

Original: Representation
Revised: Representation

Original: Dim
Revised: Dim

Original: URL
Revised: URL

Original: Checkpoint
Revised: Checkpoint

Original: InceptionV 2048 https://github.
Revised: InceptionV 2048 https://github.

Original: com/mseitzer/ pytorch-fid https://github.com/ mseitzer/pytorch-fid/ releases/download/fid_ weights/pt_inception-201512-05-6726825d.pth
Revised: com/mseitzer/ pytorch-fid https://github.com/ mseitzer/pytorch-fid/ releases/download/fid_ weights/pt_inception-201512-05-6726825d.pth

Original: Resnet2048 https://pytorch.
Revised: Resnet2048 https://pytorch.

Original: org/docs/stable/ torchvision/
Revised: org/docs/stable/ torchvision/

Original: models.html https://download.pytorch.
Revised: models.html https://download.pytorch.

Original: org/models/resnet5019c8e357.pth
Revised: org/models/resnet5019c8e357.pth

Original: Imagenet21k 2048 https://tfhub.
Revised: Imagenet21k 2048 https://tfhub.

Original: dev/google/bit/mr50x1/ https://tfhub.dev/google/ bit/m-r50x1/
Revised: dev/google/bit/mr50x1/ https://tfhub.dev/google/ bit/m-r50x1/

Original: SwAV 2048 https:// github.com/ facebookresearch/ swav https://dl.fbaipublicfiles.
Revised: SwAV 2048 https:// github.com/ facebookresearch/ swav https://dl.fbaipublicfiles.

Original: com/deepcluster/swav_800ep_ pretrain.pth.tar
Revised: com/deepcluster/swav_800ep_ pretrain.pth.tar

Original: DeepClusterV2 2048 https:// github.com/ facebookresearch/ swav https://dl.fbaipublicfiles.
Revised: DeepClusterV2 2048 https:// github.com/ facebookresearch/ swav https://dl.fbaipublicfiles.

Original: com/deepcluster/ deepclusterv2_800ep_ pretrain.pth.tar
Revised: com/deepcluster/ deepclusterv2_800ep_ pretrain.pth.tar

Revised: MoCoV 2048 https:// github.com/ facebookresearch/ moco https://dl.fbaipublicfiles.

Revised: com/moco/moco_checkpoints/ moco_v2_800ep/moco_v2_ 800ep_pretrain.pth.tar