Original: O N S ELF -S UPERVISED I MAGE R EPRESENTATIONS FOR GAN E VALUATION
Revised: O N S ELF -S UPERVISED I MAGE R EPRESENTATIONS FOR GAN E VALUATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The embeddings from CNNs pretrained on Imagenet classiﬁcation are de-facto standard image representations for assessing GANs via FID, Precision and Recall measures.
Revised: The embeddings from CNNs pretrained on Imagenet classiﬁcation are de-facto standard image representations for assessing GANs via FID, Precision and Recall measures.

Original: Despite broad previous criticism of their usage for non-Imagenet domains, these embeddings are still the top choice in most of the GAN literature.
Revised: Despite broad previous criticism of their usage for non-Imagenet domains, these embeddings are still the top choice in most of the GAN literature.

Original: In this paper, we advocate the usage of the state-of-the-art self-supervised representations to evaluate GANs on the established non-Imagenet benchmarks.
Revised: In this paper we advocate using state-of-the-art self supervised representations to evaluate GANs on the established non-Imagenet benchmarks.

Original: These representations, typically obtained via contrastive learning, are shown to provide better transfer to new tasks and domains, therefore, can serve as more universal embeddings of natural images.
Revised: These representations typically obtained via contrastive or clustering based approaches provide better transfer to new tasks and domains therefore, can serve as more universal embeddings of natural images

Original: With extensive comparison of the recent GANs on the common datasets, we show that self-supervised representations produce a more reasonable ranking of models in terms of FID/Precision/Recall, while the ranking with classiﬁcation-pretrained embeddings often can be misleading.
Revised: With extensive comparison of the recent GANs on the standard datasets , we demonstrate that self-supervised representations pro duce a more reasonable ranking of models in terms of FID/ Precision /Recall while the ranking with classiﬁcation-pretrained embeddings often can be misleading.

Revised: Furthermore, using self-supervised representations often improves the sampleefﬁciency of FID, which makes it more reliable in limited-data regimes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial networks (GANs) are an extremely active research direction in machine learning.
Revised: Generative adversarial networks (GANs) are an extremely active research direction in machine learning.

Original: The intensive development of the ﬁeld requires established quantitative measures to assess constantly appearing models.
Revised: The intensive development of the ﬁeld requires established quantitative measures to assess constantly appearing models.

Original: While a large number of evaluation protocols were proposed (Borji, 2019; Xu et al., 2018; Zhou et al., 2019; Naeem et al., 2020), there is still no consensus regarding the best evaluation measure.
Revised: While a large number of evaluation protocols were proposed (Borji, 2019; Xu et al., 2018; Zhou et al., 2019; Naeem et al., 2020), there is still no consensus regarding the best evaluation measure.

Original: Across the existing measures, the Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) and Precision/Recall (Kynk ¨ a ¨ anniemi et al., 2019) are the most widely adopted due to their simplicity and decent consistency with human judgments.
Revised: Across the existing measures, the Fr´echet Inception Distance (FID) (Heusel et al., 2017) and Precision/Recall (Kynk¨a¨anniemi et al., 2019) are the most widely adopted due to their simplicity and decent consistency with human judgments.

Original: FID and Precision/Recall quantify the discrepancy between distributions of real and generated images.
Revised: FID and Precision/Recall quantify the discrepancy between distributions of real and generated images.

Original: Since these distributions are complicated to describe in the original RGB space, the images are represented by embeddings, typically extracted with CNNs pretrained on the Imagenet classiﬁcationtask (Deng et al., 2009).
Revised: Since these distributions are complicated to describe in the original RGB space, the images are represented by embeddings, typically extracted with CNNs pretrained on the Imagenet classiﬁcation (Deng et al., 2009).

Original: While FID computed with these embeddings was shown to correlate with human evaluation (Heusel et al., 2017), these observations were mostly obtained onthe datasets, semantically close to Imagenet.
Revised: While FID computed with these embeddings was shown to correlate with human evaluation (Heusel et al., 2017), these observations were mostly obtained on datasets, semantically close to Imagenet.

Original: Meanwhile, on non-Imagenet datasets, these embeddings can provide inadequate evaluation, as widely reported in the literature (Rosca et al., 2017; Barratt & Sharma, 2018;
Revised: Meanwhile, on non-Imagenet datasets, FID can result in inadequate evaluation, as widely reported in the literature (Rosca et al., 2017; Barratt & Sharma, 2018;

Original: Zhou et al., 2019).
Revised: Zhou et al., 2019).

Original: In this work, we propose to employ the state-of-the-art self-supervised models (Chen et al., 2020a;
Revised: In this work, we propose to employ the state-of-the-art self-supervised models (Chen et al., 2020a;

Original: He et al., 2020; Caron et al., 2020) to extract image embeddings for GAN evaluation.
Revised: He et al., 2020; Caron et al., 2020) to extract image embeddings for GAN evaluation.

Original: These models were shown to produce features that transfer better to new tasks, hence, they become a promising candidate to provide a more universal representation.
Revised: These models were shown to produce features that transfer better to new tasks, hence, they become a promising candidate to provide a more universal representation.

Original: Intuitively, classiﬁcation-pretrained embeddings by design can suppress the information, irrelevant for the Imagenet class labels, which, however, can be crucial for other domains, like human faces.
Revised: Intuitively, classiﬁcation-pretrained embeddings by design can suppress the information, irrelevant for the Imagenet class labels, which, however, can be crucial for other domains, like human faces.

Original: On the contrary, self-supervised models, mostly trained via contrastive  learning, do not have such a bias since their main goal is typically to learn invariances to common image augmentations.
Revised: On the contrary, self-supervised models, mostly trained via contrastive or clustering-based learning, do not have such a bias since their main goal is typically to learn invariances to common image augmentations.

Original: To justify the usage of self-supervised embeddings, we perform a thorough comparison of the recent GAN models trained on the ﬁve most common benchmark datasets.
Revised: To justify the usage of self-supervised embeddings, we perform a thorough comparison of the recent GAN models trained on the ﬁve most common benchmark datasets.

Original: We demonstrate that classiﬁcation-pretrained embeddings can lead to incorrect ranking in terms of FID, Precision, and
Revised: We demonstrate that classiﬁcation-pretrained embeddings can lead to incorrect ranking in terms of FID, Precision, and

Original: Recall, which are the most popular metrics.
Revised: Recall, which are the most popular metrics.

Original: On the other hand, self-supervised representations produce more reasonable ranking, advocating their advantage over “classiﬁcation-oriented” counterparts.
Revised: On the other hand, self-supervised representations produce more sensible ranking, advocating their advantage over “classiﬁcation-oriented” counterparts.

Original: Since all the checkpoints needed to compute self-supervised embeddings are publicly available, they can serve as a handy instrument for GAN comparison, consistent between different papers.
Revised: Since all the checkpoints needed to compute self-supervised embeddings are publicly available, they can serve as a handy instrument for GAN comparison, consistent between different papers.

Revised: We release the code for the “self-supervised” GAN evaluation along with data and human labeling reported in the paper online 1 .

Original: Overall, the contributions of this paper aresummarized as follows:
Revised: To sum up, the contributions of this paper are as follows:

Original: To the best of our knowledge, our work is the ﬁrst to employ self-supervised image representationsobtained via contrastive learning to evaluate GANs trained on natural images.
Revised: To the best of our knowledge, our work is the ﬁrst to employ self-supervised image representations to evaluate GANs trained on natural images.

Original: By extensive experiments on the standard non-Imagenet benchmarks, we demonstrate that the usage of self-supervised representations provides  more accurate GAN comparison.
Revised: By extensive experiments on the standard non-Imagenet benchmarks, we demonstrate that the usage of self-supervised representations provides a more reliable GAN comparison.

Original: We have evaluated a large number of publicly available GANs in terms of “self-supervised” FID, Precision, and Recall, providing these values for further research as a reference.
Revised: We show that the FID measure computed with self-supervised representations often has higher sample-efﬁciency and analyze the sources of this advantage.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: GAN evaluation measures.
Revised: GAN evaluation measures.

Original: Over the last years, a variety of quantitative GAN evaluation methods have been developed by the community, and the development process has yet to converge since all the measures possess speciﬁc weaknesses (Borji, 2019;
Revised: Over the last years, a variety of quantitative GAN evaluation methods have been developed by the community, and the development process has yet to converge since all the measures possess speciﬁc weaknesses (Borji, 2019;

Original: Xu et al., 2018).
Revised: Xu et al., 2018).

Original: The Inception Score (Salimans et al., 2016) was the ﬁrst widely adopted measure but was shown to be hardly applicable for non-Imagenet domains (Barratt & Sharma, 2018).
Revised: The Inception Score (Salimans et al., 2016) was the ﬁrst widely adopted measure but was shown to be hardly applicable for non-Imagenet domains (Barratt & Sharma, 2018).

Original: The Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) quantiﬁes dissimilarity of real and generated distributions, computing the Wasserstein distance between their Gaussian approximations, and is currently the most popular scalar measure of GAN’s quality.
Revised: The Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) quantiﬁes dissimilarity of real and generated distributions, computing the Wasserstein distance between their Gaussian approximations, and is currently the most popular scalar measure of GAN’s quality.

Original: Several recent measures were proposed (Sajjadi et al., 2018; Kynk¨a¨anniemi et al., 2019; Naeem et al., 2020) that separately evaluate ﬁdelity and diversity of GAN-produced images.
Revised: Several recent measures were proposed (Sajjadi et al., 2018; Kynk ¨ a ¨ anniemi et al., 2019; Naeem et al., 2020) that separately evaluate ﬁdelity and diversity of GAN-produced images.

Original: All of them mostly use the embeddings produced by the Imagenet classiﬁcation CNN.
Revised: All of them mostly use the embeddings produced by the Imagenet classiﬁcation CNN.

Original: A recent work (Zhou et al., 2019) has introduced a human-in-the-loop measure, which is more reliable compared to automated ones but cannot be used, e.g., for monitoring the training process.
Revised: A recent work (Zhou et al., 2019) has introduced a human-in-the-loop measure, which is more reliable compared to automated ones but cannot be used, e.g., for monitoring the training process.

Original: We focus on three the most widely used measures: FID, Precision, and Recall, which are discussed brieﬂy below.
Revised: We focus on three the most widely used measures: FID, Precision, and Recall, which are discussed brieﬂy below.

Original: Fr´echet Inception Distance quantiﬁes the discrepancy between the distributions of real and generated images, denoted by p D and p G .
Revised: Fr´echet Inception Distance quantiﬁes the discrepancy between the distributions of real and generated images, denoted by p D and p G .

Original: Both p D and p G are deﬁned on the high-dimensional image space forming nontrivial manifolds, which are challenging to approximate by simple functions.
Revised: Both p D and p G are deﬁned on the high-dimensional image space forming nontrivial manifolds, which are challenging to approximate by simple functions.

Original: To be practical, FID operates in the lower-dimensional space of image embeddings.
Revised: To be practical, FID operates in the lower-dimensional space of image embeddings.

Original: Formally, the embeddings are deﬁned by a map f : R N → R d , where N and d correspond to the dimensionalities of the images and embeddings spaces, respectively.
Revised: Formally, the embeddings are deﬁned by a map f : R N → R d , where N and d correspond to the dimensionalities of the images and embeddings spaces, respectively.

Original: By design, FID measures the dissimilarity between the induced distributions fp D , fp G as follows.
Revised: By design, FID measures the dissimilarity between the induced distributions fp D , fp G as follows.

Original: First, fp D and fp G are approximated by Gaussian distributions.
Revised: First, fp D and fp G are approximated by Gaussian distributions.

Original: Then the Wasserstein distance between these distributions is evaluated.
Revised: Then the Wasserstein distance between these distributions is evaluated.

Original: As was shown in (Dowson & Landau, 1982), for distributions deﬁned by the means µ D , µ G and the covariance matrices Σ D , Σ G , this quantity equals to (cid:107) µ D − µ G (cid:107) 22 + tr (Σ D + Σ G − 2(Σ D Σ G ) 12 ) .
Revised: As was shown in (Dowson & Landau, 1982), for distributions deﬁned by the means µ D , µ G and the covariance matrices Σ D , Σ G , this quantity equals to (cid:107) µ D − µ G (cid:107) 22 + tr (Σ D + Σ G − 2(Σ D Σ G ) 12 ) .

Original: Lower FID values correspond to higher similarity between p G and p D ; hence, can be used to evaluate the performance of generative models.
Revised: Lower FID values correspond to higher similarity between p G and p D ; hence, can be used to evaluate the performance of generative models.

Original: As a common practice in the FID computation, one typically uses the activations from the InceptionV3 (Szegedy et al., 2016) pretrained on Imagenet classiﬁcation.
Revised: As a common practice in the FID computation, one typically uses the activations from the InceptionV3 (Szegedy et al., 2016) pretrained on Imagenet classiﬁcation.

Original: Precision and Recall.
Revised: Precision and Recall.

Original: When assessing generative models, it is important to quantify both the visual quality of generated images and the model diversity, e.g., to diagnose mode collapsing.
Revised: When assessing generative models, it is important to quantify both the visual quality of generated images and the model diversity, e.g., to diagnose mode collapsing.

Original: However, the scalar FID values were shown (Sajjadi et al., 2018; Kynk¨a¨anniemi et al., 2019) to sacriﬁce diversity in favor of visual quality, therefore FID cannot serve as the only sufﬁcient metric.
Revised: However, the scalar FID values were shown (Sajjadi et al., 2018; Kynk ¨ a ¨ anniemi et al., 2019) to sacriﬁce diversity in favor of visual quality, therefore FID cannot serve as the only sufﬁcient metric.

Original: To this end, (Sajjadi et al., 2018) introduced Precision and Recall, which aim to measure the image realism and the model diversity, respectively.
Revised: To this end, (Sajjadi et al., 2018) introduced Precision and Recall, which aim to measure the image realism and the model diversity, respectively.

Original: A recent follow-up (Kynk¨a¨anniemi et al., 2019) elaborates on these metrics and proposes a reasonable procedure to quantify both precision and recall based only on the image embeddings.
Revised: A recent follow-up (Kynk¨a¨anniemi et al., 2019) elaborates on these metrics and proposes a reasonable procedure to quantify both precision and recall based only on the image embeddings.

Original: In a nutshell, (Kynk¨a¨anniemi et al., 2019) assumes that the visual quality of a particular sample is high if its embedding is neighboring for the embeddings of the real images.
Revised: In a nutshell, (Kynk¨a¨anniemi et al., 2019) assumes that the visual quality of a particular sample is high if its embedding is neighboring for the embeddings of the real images.

Original: On the other hand, a given real image is considered covered by the model if its embedding belongs to the neighborhood of embeddings of the generated images.
Revised: On the other hand, a given real image is considered covered by the model if its embedding belongs to the neighborhood of embeddings of the generated images.

Original: Self-supervised representations.
Revised: Self-supervised representations.

Original: Self-supervised learning is currently attracting much research attention, especially tothe contrastive learning  methods (Chen et al., 2020a;
Revised: Self-supervised learning is currently attracting much research attention, especially to contrastive learning and clustering-based methods (Chen et al., 2020a;

Original: He et al., 2020; Caron et al., 2020).
Revised: He et al., 2020; Caron et al., 2020).

Original: The common idea behind contrastive methods is to construct representations that are close for positive imagepairs and distant for negative ones, where positive pairs are produced automatically via augmentations.
Revised: The common idea behind these methods is to construct representations that are invariant to a wide range of common image augmentations.

Original: The recent self-supervised methods were shown to provide more transferrable (He et al., 2020; Caron et al., 2020) and robust (Hendrycks et al., 2019) features, which implies their usage as more universal representations.
Revised: The recent self-supervised methods were shown to provide more transferrable (He et al., 2020; Caron et al., 2020) and robust (Hendrycks et al., 2019) features, which implies their usage as more universal representations.

Original: In this paper, we show them being a better alternative compared to established classiﬁer-produced embeddings in the context of GAN assessment
Revised: In this paper, we show them being a better alternative compared to established classiﬁer-produced embeddings in the context of GAN assessment.

Original:  GAN EVALUATION
Revised:  GAN EVALUATION

Original: Here we systematically compare the publicly available GANs to highlight the cases of misleading comparison with classiﬁcation-pretrained embeddings.
Revised: Here we systematically compare the publicly available GANs to highlight the cases of misleading comparison with classiﬁcation-pretrained embeddings.

Original: Our goal is to demonstrate that in these cases, self-supervised embeddings are a better alternative, while in other cases, the rankings with both types of embeddings are mostly consistent.
Revised: Our goal is to demonstrate that selfsupervised embeddings are a better alternative in these cases, while in other cases, the rankings with both types of embeddings are mostly consistent.

Original: We examine GANs trained on ﬁve popular benchmarks:
Revised: We examine open-sourced GAN models 2 trained on ﬁve popular benchmarks:

Original: • CelebaHQ 1024x1024 (Karras et al., 2017) with the following GAN models: StyleGAN without truncation  (Karras et al., 2019a), MSG (Karnewar & Wang, 2020) with truncation 0 .
Revised: • CelebaHQ 1024x1024 (Karras et al., 2017) with the following GAN models: StyleGAN with truncation 0 . 7 (Karras et al., 2019a) and without it, MSG (Karnewar & Wang, 2020) with truncation 0 .

Original: 6 and without it, PGGAN (Karras et al., 2017).
Revised: 6 and without it, PGGAN (Karras et al., 2017).

Original: To compute the metrics, we use 30k real and synthetic images; • FFHQ 1024x1024 (Karras et al., 2019a) with the following GAN models: StyleGAN (Karras et al., 2019a), StyleGAN2 (Karras et al., 2019b), MSG (Karnewar & Wang, 2020) with truncation 0 .
Revised: To compute the metrics, we use 30k real and synthetic images; • FFHQ 1024x1024 (Karras et al., 2019a) with the following GAN models: StyleGAN (Karras et al., 2019a), StyleGAN2 (Karras et al., 2019b), MSG (Karnewar & Wang, 2020) with truncation 0 .

Original: 6 and without it.
Revised: 6 and without it.

Original: To compute the metrics, we use 30k real and synthetic images; • LSUN Bedroom 256x

Revised: To compute the metrics, we use 30k real and synthetic images; • LSUN Bedroom 256x

Original: (Yu et al., 2015) with the following GAN models: StyleGAN (Karras et al., 2019a), PGGAN (Karras et al., 2017),
Revised: (Yu et al., 2015) with the following GAN models: StyleGAN (Karras et al., 2019a) with truncation 0 .

Original:  COCO-GAN
Revised: 7 and without it, PGGAN (Karras et al., 2017), COCO-GAN

Original: (Lin et al., 2019), RPGAN (Voynov & Babenko, 2019), RPGAN with high diversity (RPGAN div.).
Revised: (Lin et al., 2019), RPGAN (Voynov & Babenko, 2019), RPGAN with high diversity (RPGAN div.).

Original: RPGAN generates 128x128 images, so we upscale them to 256x256.
Revised: RPGAN generates 128x128 images, so we upscale them to 256x256.

Original: To compute the metrics, we use 30k real and synthetic images; • LSUN Church 256x

Revised: To compute the metrics, we use 30k real and synthetic images; • LSUN Church 256x

Original: (Yu et al., 2015) with the models: StyleGAN2 (Karras et al., 2019b) with truncation 0 .
Revised: (Yu et al., 2015) with the models: StyleGAN2 (Karras et al., 2019b) with truncation 0 .

Original: 5 and without it, MSG (Karnewar & Wang, 2020) with truncation 0 .
Revised: 5 and without it, MSG (Karnewar & Wang, 2020) with truncation 0 .

Original: 6 and without it, PGGAN (Karras et al., 2017), SNGAN (Miyato et al., 2018).
Revised: 6 and without it, PGGAN (Karras et al., 2017), SNGAN (Miyato et al., 2018).

Original: SNGAN generates 128x128 images, so we upscale them to 256x256.
Revised: SNGAN generates 128x128 images, so we upscale them to 256x256.

Original: To compute the metrics, we use 100k real and synthetic images; • Imagenet 128x

Revised: To compute the metrics, we use 100k real and synthetic images; • Imagenet 128x

Original: (Deng et al., 2009) with the following GAN models: BigGAN (Brock et al., 2019), BigGAN-deep
Revised: (Deng et al., 2009) with the following GAN models: BigGAN (Brock et al., 2019), BigGAN-deep

Original: (Brock et al., 2019), S3GAN (Lucic et al., 2019), Your Local GAN (YLG) (Daras et al., 2020).
Revised: (Brock et al., 2019), S3GAN (Lucic et al., 2019), Your Local GAN (YLG) (Daras et al., 2020).

Original: To compute the metrics, we use 50k real and synthetic images (50 per class);.
Revised: To compute the metrics, we use 50k real and synthetic images (50 per class);.

Original: We include this dataset to demonstrate that for Imagenet, the proposed self-supervised representations provide consistent ranking with commonly used InceptionV3 embeddings.
Revised: We include this dataset to demonstrate that for Imagenet, the proposed self-supervised representations provide consistent ranking with commonly used InceptionV3 embeddings.

Original: To compute image embeddings we use the following publicly available models:
Revised: To compute image embeddings, we use the following publicly available models:

Original: • InceptionV3 (Szegedy et al., 2016) pretrained on the ILSVRC-2012 task (Deng et al., 2009); • Resnet50 (He et al., 2016) pretrained on the ILSVRC-2012 task.
Revised: • InceptionV3 (Szegedy et al., 2016) pretrained on the ILSVRC-2012 task (Deng et al., 2009); • Resnet50 (He et al., 2016) pretrained on the ILSVRC-2012 task.

Original: We include this model since self-supervised models employ Resnet50, therefore, it is important to demonstrate that better GAN ranking comes from the training objective rather than the deeper architecture;
Revised: We include this model since self-supervised models employ Resnet50, therefore, it is important to demonstrate that better GAN ranking comes from the training objective rather than the deeper architecture;

Original: • Imagenet21k
Revised: • Imagenet21k

Original: (Kolesnikov et al., 2019) pretrained on the multi-label classiﬁcation task on approximately 14M images from the full Imagenet.
Revised: (Kolesnikov et al., 2019) pretrained on the multi-label classiﬁcation task on approximately 14M images from the full Imagenet.

Original: Kolesnikov et al.
Revised: Kolesnikov et al.

Original: (2019) have shown that supervised pretraining on huge datasets provides more transferrable features, therefore, Imagenet21k can also potentially provide more universal representations.
Revised: (2019) have shown that supervised pretraining on huge datasets provides more transferrable features, therefore, Imagenet21k can also potentially provide more universal representations.

Original: The model architecture is Resnet50;  • SwAV (Caron et al., 2020) is the state-of-the-art self-supervised image representation model trained on ILSVRC-2012.
Revised: The model architecture is Resnet50; [Table] • SwAV (Caron et al., 2020) is the state-of-the-art self-supervised image representation model trained on ILSVRC-2012.

Original: The idea of SwAV is to simultaneously cluster the images while enforcing consistency between cluster assignments produced for different augmentations of the same image.
Revised: The idea of SwAV is to simultaneously cluster the images while enforcing consistency between cluster assignments produced for different augmentations of the same image.

Original: The model architecture is Resnet50; • DeepClusterV2 (Caron et al., 2020) is another self-supervised model obtained by alternating between pseudo-labels generation via k-means clustering and training the network with a classiﬁcation loss supervised by these pseudo-labels.
Revised: The model architecture is Resnet50; • DeepClusterV2 (Caron et al., 2020) is another self-supervised model obtained by alternating between pseudo-labels generation via k-means clustering and training the network with a classiﬁcation loss supervised by these pseudo-labels.

Original: The model architecture is Resnet50; • MoCoV2 (Chen et al., 2020b) is the state-of-the-art contrastive learning approach, which training objective enforces the closeness of representations produced for different augmentations of the same image while pushing apart the representations of unrelated images.
Revised: The model architecture is Resnet50; • MoCoV2 (Chen et al., 2020b) is the state-of-the-art contrastive learning approach, which training objective enforces the closeness of representations produced for different augmentations of the same image while pushing apart the representations of unrelated images.

Original: The model architecture is Resnet50.
Revised: The model architecture is Resnet50.

Original: Three self-supervised models listed above outperform supervised pretraining on a number of transfer tasks (He et al., 2020; Caron et al., 2020), which implies that their embeddings capture more information relevant for these tasks, compared to supervised models pretrained on Imagenet.
Revised: Three self-supervised models listed above outperform supervised pretraining on a number of transfer tasks (He et al., 2020; Caron et al., 2020), which implies that their embeddings capture more information relevant for these tasks, compared to supervised models pretrained on Imagenet.

Original: Below, for 
Revised: Below, for a large number of publicly available GANs, we present the values of FID, Precision, and Recall met-

Original: [Table]
Revised: [Table]

Original: a large number of publicly available GANs, we present the values of FID, Precision, and Recall metrics computed with different embeddings.
Revised: rics computed with different embeddings.

Original: For the cases where the GANs ranking is inconsistent, we aim to show that the ranking obtained with the self-supervised representations is more reasonable.
Revised: For the cases where the GANs ranking is inconsistent, we aim to show that the ranking obtained with the self-supervised representations is more reasonable.

Original: 3. F R ´ ECHET I NCEPTION D ISTANCE
Revised: 3. F R ´ ECHET I NCEPTION D ISTANCE

Original: The FID values for the non-Imagenet datasets computed with different embeddings are shown in
Revised: The FID values for the non-Imagenet datasets computed with different embeddings are shown in

Original: Table 1.
Revised: Table 1.

Original: The cases of inconsistent ranking with supervised InceptionV3 and self-supervised SwAV embeddings are highlighted in color.
Revised: The cases of inconsistent ranking with supervised InceptionV3 and self-supervised SwAV embeddings are highlighted in color.

Original: The key observations are listed below: (a) On the CelebaHQ dataset, SwAV ranks StyleGAN higher, while InceptionV3/Resnet50 prefer 
Revised: The key observations are listed below: (a) On CelebaHQ, SwAV ranks StyleGAN* higher, while InceptionV3/Resnet50 prefer PGGAN.

Original: PGGAN. Figure 1 shows random samples from both StyleGAN and PGGAN and clearly demonstrates the superiority of StyleGAN.
Revised: Figure 1 shows random samples from both StyleGAN* and PGGAN and clearly demonstrates the superiority of StyleGAN*.

Original: To investigate the reasons why SwAV produces a more adequate ranking compared to Inception/Resnet50, we perform two additional experiments.
Revised: To investigate the reasons why SwAV produces a more adequate ranking compared to Inception/Resnet50, we perform two additional experiments.

Original: (I)
Revised: (I)

Original: First, we verify that SwAV embeddings capture more information relevant for face images.
Revised: First, we verify that SwAV embeddings capture more information relevant for face images.

Original: The Celeba dataset (Liu et al., 2018) provides labels of 40 attributes for each image, describing various person properties (gender, age, hairstyle, etc.).
Revised: The Celeba dataset (Liu et al., 2018) provides labels of 40 attributes for each image, describing various person properties (gender, age, hairstyle, etc.).

Original: For each attribute, we train 4-layer feedforward neural network[Table] with 2048 neurons on each layer with cross-entropy loss, which learns to predict the attribute from the SwAV/Inception embedding.
Revised: For each attribute, we train 4-layer feedforward neural network with 2048 neurons on each layer with cross-entropy loss, which learns to predict the attribute from the SwAV/Inception embedding.

Original: For all attributes, the predictions from SwAV embeddings appear to be more accurate compared to InceptionV3 (several examples are given in Table 2).
Revised: For all attributes, the predictions from SwAV embeddings appear to be more accurate compared to InceptionV3 (several examples are given in Table 2).

Original: It conﬁrms the intuition that InceptionV3 representations partially suppress the information about small facial details, which, however, is critical to identify more realistic images.
Revised: It conﬁrms the intuition that InceptionV3 representations partially suppress the information about small facial details, which, however, is critical to identify more realistic images.

Original: (II)
Revised: (II)

Original: As a qualitative experiment, we compare SwAV and supervised Resnet50 embeddings visually via a recent technique described in Rombach et al.
Revised: As a qualitative experiment, we compare SwAV and supervised Resnet50 embeddings visually via a recent technique described in Rombach et al.

Original: In a nutshell, this technique reveals the invariances learned by the particular representation model: for a given image, it visualizes several images, having approximately the same embedding.
Revised: In a nutshell, this technique reveals the invariances learned by the particular representation model: for a given image, it visualizes several images having approximately the same embedding.

Original: By inspecting these images, one can analyze what factors of variations are not captured in the embedding (see the details in Section A.1).
Revised: By inspecting these images, one can analyze what factors of variations are not captured in the embedding (see the details in Section A.2).

Original: Two illustrative examples of such visualization for SwAV and Resnet50 are shown in Figure 2, demonstrating that Resnet50 embeddings are more invariant to sensitive information, like gender or race, compared to SwAV.
Revised: Two illustrative examples of such visualization for SwAV and Resnet50 are shown in Figure 2, demonstrating that Resnet50 embeddings are more invariant to sensitive information, like gender or race, compared to SwAV.

Original: Such ignorance of sensitive information makes supervised embeddings less appealing to use as universal representations.
Revised: Such ignorance of sensitive information makes supervised embeddings less appealing to use as universal representations.

Original: One of the key ingredients of the visualization method is an autoencoder, which is expected to capture all relevant information from an image.
Revised: One of the key ingredients of the visualization method is an autoencoder, which is expected to capture all relevant information from an image.

Original: However, we argue that autoencoder representations are not well-suited for evaluating generative models and elaborate on this in detail in Section D.
Revised: However, we argue that autoencoder representations are not well-suited for evaluating generative models and elaborate on this in detail in Section D.

Original: (b) On the Bedroom dataset, there are two inconsistencies in InceptionV3 and SwAV ranking.
Revised: (b) On Bedroom, there are two inconsistencies in InceptionV3 and SwAV ranking.

Original: The ﬁrst is that SwAV ranks StyleGAN higher than PGGAN and the second is that SwAV ranks RPGAN higher than COCO-GAN.
Revised: The ﬁrst is that SwAV ranks StyleGAN higher than PGGAN and the second is that SwAV ranks RPGAN higher than COCO-GAN.

Original: Figure 3 shows the samples from StyleGAN, PGGAN, RPGAN, and COCO-GAN and demonstrates that the ranking according to SwAV embeddings is more adequate.
Revised: Figure 3 shows the samples from StyleGAN, PGGAN, RPGAN, and COCOGAN and demonstrates that the ranking according to SwAV embeddings is more adequate.

Original: Namely, the quality of StyleGAN-generated images is substantially higher.
Revised: Namely, the quality of StyleGAN-generated images is substantially higher.

Original: Also, it is difﬁcult to identify a favorite among RPGAN and COCO-GAN visually, while InceptionV3 embeddings claim strong superiority of the COCO-GAN model.
Revised: Also, it is difﬁcult to identify a favorite among RPGAN and COCO-GAN visually, while InceptionV3 embeddings claim strong superiority of the COCO-GAN model.

Original: On the other hand, self-supervised embeddings consider these models as comparable, which is better aligned with human perception.c)
Revised: On the other hand, self-supervised embeddings consider these models as comparable, which is better aligned with human perception.

Original: There is also a case of the inconsistent ranking of Church models, namely MSG and PGGAN. But since the difference of the FID values for MSG and PGGAN is small for both InceptionV3 and
Revised: [Figure]

Original: SwAV, we do not consider it as a strong disagreement. (d)
Revised: [Figure]

Original: The rankings with the supervised InceptionV3/Resnet50 embeddings are consistent with each other, whilebeing different from the rankings with the self-supervised ones.
Revised: The rankings with the supervised InceptionV3/Resnet50 embeddings are consistent with each other while different from the rankings with the self- ones.

Original: It indicates that themodel architecture does not affect GAN ranking, and it is the training objective that matters.
Revised: the architecture does not affect GAN ranking, and it is the training objective that matters.

Original: e) Imagenet21k
Revised: (e)

Original:  corrects some cases of misleading ranking with InceptionV3, but not all of them.
Revised: Imagenet21k corrects some cases of misleading ranking with InceptionV3, but not all of them.

Original: Namely, it correctly ranks StyleGAN and PGGAN on Bedroom, while being wrong on CelebaHQ.
Revised: Namely, it correctly ranks StyleGAN and PGGAN on Bedroom while being wrong on CelebaHQ.

Original: (f) SwAV and DeepClusterV2 have minimal inconsistencies in the ranking of MSG* vs PGGAN on
Revised: (f) SwAV and DeepClusterV2 have minimal inconsistencies in the ranking of MSG* vs PGGAN on

Original: CelebaHQ and StyleGAN2* vs SNGAN on Church, but the differences in the absolute values of the
Revised: CelebaHQ and StyleGAN2* vs SNGAN on Church, but the differences in the absolute values of the

Original: FID metric are negligible, so we consider these embedding models as mostly consistent.
Revised: FID metric are negligible, so we consider these embedding models as mostly consistent.

Original: (g) MoCoV2 ﬁxes some ranking mistakes with InceptionV3, but not all of them.
Revised: (g) MoCoV2 ﬁxes some ranking mistakes with InceptionV3, but not all of them.

Original: While it ﬁxes the ranking of StyleGAN and PGGAN on Bedroom and reduces the gap between RPGAN and COCOGAN, the ranking of PGGAN and StyleGAN on CelebaHQ is still incorrect.
Revised: While it ﬁxes the ranking of StyleGAN and PGGAN on Bedroom and reduces the gap between RPGAN and COCOGAN, the ranking of PGGAN and StyleGAN on CelebaHQ is still incorrect.

Original: Overall, the most reasonable rankings are obtained using SwAV/DeepCluster, which have signiﬁcantly higher transfer 
Revised: Overall, the most reasonable rankings are obtained using SwAV/DeepCluster, which have signiﬁcantly higher transfer performance compared to MoCoV2.

Original: [Figure]
Revised: In further experiments, we focus on the most transferable

Original: generated by StyleGAN is substantially higher. [Figure]
Revised: SwAV/DeepCluster models.

Original: Overall, self-supervised embeddings provide a more reasonable FID ranking across existing nonImagenet benchmarks.
Revised: Overall, self-supervised embeddings provide a more reasonable FID ranking across existing nonImagenet benchmarks.

Original: For completeness, we also report the FID values for the Imagenet dataset in
Revised: For completeness, we also report the FID values for the Imagenet dataset in

Original: Table 6.
Revised: Table 6.

Original: In this case, rankings with all embeddings are the same, which conﬁrms that the SwAV representations can be used for Imagenet as well, while it is not the main focus of our work. 
Revised: In this case, rankings with all embeddings are the same, which conﬁrms that the SwAV representations can be used for Imagenet as well, while it is not the main focus of our work. [Figure] [Table]

Original: 3. P RECISION
Revised: 3. P RECISION

Original: The values of the Precision metric are reported in Table 3.
Revised: The values of the Precision metric are reported in Table 3.

Original: The main observations are listed below: (a) As with FID, all supervised InceptionV3/Resnet50 embeddings provide the same ranking, except minor differences between MSG with truncation and StyleGAN on CelebaHQ, and StyleGAN and PGGAN on Church.
Revised: The main observations are listed below: (a) As with FID, all supervised InceptionV3/Resnet50 embeddings provide the same ranking, except minor differences between MSG with truncation and StyleGAN on CelebaHQ, and StyleGAN2 and PGGAN on Church.

Original: Self-supervised SwAV and DeepClusterV2 are also consistent except  the negligible difference in the ranking of PGGAN and MSG on CelebaHQ and Church; (b) The most notable inconsistency between supervised and self-supervised embeddings is revealed on the LSUN-Church dataset, where InceptionV3 considers MSG to be comparable to StyleGAN2, while SwAV ranks StyleGAN2 signiﬁcantly higher. 
Revised: Self-supervised SwAV and DeepClusterV2 are also consistent except for the negligible difference in the ranking of PGGAN and MSG on CelebaHQ and Church; (b) The most notable inconsistency between supervised and self-supervised embeddings is revealed on LSUN-Church, where InceptionV3 considers MSG to be comparable to StyleGAN2, while SwAV ranks StyleGAN2 signiﬁcantly higher. (I) To

Original: (I) To analyze which ranking of two GANs is more reasonable, we perform the following.
Revised: analyze which ranking of two GANs is more reasonable, we perform the following.

Original: On the synthetic data from the ﬁrst GAN, we train a classiﬁer that aims to distinguish between real and synthetic images.
Revised: On the synthetic data from the ﬁrst GAN, we train a classiﬁer that aims to distinguish between real and synthetic images.

Original: This classiﬁer is then evaluated on the synthetic data from the second GAN.
Revised: This classiﬁer is then evaluated on the synthetic data from the second GAN.

Original: Concretely, we train a classiﬁer to detect synthetic images on real LSUN-Church and the images generated by MSG.
Revised: Concretely, we train a classiﬁer to detect synthetic images on real LSUN-Church and the images generated by MSG.

Original: Then we evaluate this model on hold-out real images and images produced by StyleGAN2.
Revised: Then we evaluate this model on hold-out real images and images produced by StyleGAN2.

Original: Intuitively, if a model was trained on high-quality synthetic samples, it will easily detect lower-quality ones.
Revised: Intuitively, if a model was trained on high-quality synthetic samples, it will easily detect lower-quality ones.

Original: On the other hand, if the model learns to detect only low-quality synthetics, it will be harder to discriminate real images from[Figure] [Table] the high-quality one.
Revised: On the other hand, if the model learns to detect only low-quality synthetics, it will be harder to discriminate real images from the high-quality ones.

Original: In this experiment [Table] we employ a Resnet50 classiﬁer with a binary cross-entropy loss.
Revised: In this experiment, we employ a Resnet50 classiﬁer with a binary cross-entropy loss.

Original: The results forthe Churchdataset are provided in Table 4, meaning that the StyleGAN2 images are of higher quality, therefore, the SwAV ranking is more reasonable.
Revised: The results for Church are provided in Table 4, meaning that the StyleGAN2 images are of higher quality, therefore, the SwAV ranking is more reasonable.

Original: (II)
Revised: (II)

Original: We also conduct a human study to determine which of the generative models gives more realistic images in terms of human perception.
Revised: We also conduct a human study to determine which of the generative models gives more realistic images in terms of human perception.

Original: For each generative model, we show ten assessors a real or randomly generated (fake) image and ask them to choose whether it is real or fake.
Revised: For each generative model, we show ten assessors a real or randomly generated (fake) image and ask them to [Table] * PGGAN COCO-GANRPGANRPGAN div. SwAVStyleGANStyleGAN * PGGAN COCO-GANRPGANRPGAN div.

Original: The error rate reﬂects the visual quality of the generative model.
Revised: DeepClusterV2StyleGANStyleGAN * PGGAN COCO-GANRPGANRPGAN div. LSUN-CHURCH InceptionV3StyleGAN2 MSG PGGANMSG * SNGAN StyleGAN2 * 0. Resnet50StyleGAN2PGGAN MSG MSG * SNGAN StyleGAN2 * 0.

Revised: Imagenet21kStyleGAN2 MSG PGGANMSG * StyleGAN2 * SNGAN 0.

Revised: SwAVStyleGAN2PGGAN MSG MSG *

Revised: StyleGAN2 * SNGAN 0.

Revised: DeepClusterV2StyleGAN2PGGAN MSG MSG * SNGAN StyleGAN2 * 0. choose whether it is real or fake.

Revised: The error rate reﬂects the visual quality of the generative model.

Original: For both models, MSG and StyleGAN2, we demonstrate to assessorsimages and the error rate is 0 .
Revised: For both models, MSG and StyleGAN2, we demonstrate to assessors 500 images, and the error rate is 0 .

Original: 4% for MSG and 2 .
Revised: 4% for MSG and 2 .

Original: 8% for StyleGAN2, which clearly shows the superiority of StyleGAN2. 
Revised: 8% for StyleGAN2, which clearly shows the superiority of StyleGAN2. producing the test

Original: [Table]
Revised: [Table]

Original: (c) Imagenet21k ranks GANs less reliably compared to SwAV.
Revised: (c) Imagenet21k ranks GANs less reliably compared to SwAV.

Original: The most notable mistake is a ranking of COCO-GAN and PGGAN on Bedroom, where PGGAN produces more visually appealing images, see Figure 3.
Revised: The most notable mistake is a ranking of COCO-GAN and PGGAN on Bedroom, where PGGAN produces more visually appealing images, see Figure 3.

Original: Another case is comparison on CelebaHQ, where Imagenet21k ranks PGGAN higher that the more powerful MSG, see samples in Section B.
Revised: Another case is comparison on CelebaHQ, where Imagenet21k ranks PGGAN higher than the more powerful MSG, see samples in Section B.

Original: 3. R ECALL
Revised: 3. R ECALL

Original: The values of the Recall metric are shown in Table 5, and the main observations are provided below:[Table]
Revised: The values of the Recall metric are shown in Table 5, and the main observations are provided below:

Original: a As in previous experiments, there are only minor inconsistencies between supervised InceptionV3 and Resnet50 models, namely, StyleGAN vs COCO-GAN onthe Bedroomdataset and MSG vs PGGAN on the Church dataset.
Revised: (a) As in previous experiments, there are only minor inconsistencies between supervised InceptionV3 and Resnet50 models, namely, StyleGAN vs COCO-GAN on Bedroom and MSG vs PGGAN on Church.

Original: The only insigniﬁcant difference between the self-supervised methods is the ranking of StyleGAN with truncation vs SNGAN on the Church dataset, however, Recall values for both models are negligible.
Revised: The only insigniﬁcant difference between the self-supervised methods is the ranking of StyleGAN with truncation vs SNGAN on Church, however, Recall values for both models are negligible.

Original: In terms of Recall, Imagenet21k ranking always coincides with the ranking obtained by self-supervised methods, except  the negligible discrepancy between MSG and PGGAN onthe Church dataset; [Table]
Revised: In terms of Recall, Imagenet21k ranking always coincides with the ranking obtained by self-supervised methods, except for the negligible discrepancy between MSG and PGGAN on

Original:  The absolute Recall values for SwAV/DeepClusterV2 are smaller compared to InceptionV3/Resnet50.
Revised: Church; (b) The absolute Recall values for SwAV/DeepClusterV2 are smaller compared to InceptionV3/Resnet50.

Original: We attribute this behavior to the fact that GANs tend to simplify images omitting the details (Bau et al., 2019), e.g., people in front of buildings, cars, fences, etc.
Revised: We attribute this behavior to the fact that GANs tend to simplify images omitting the details (Bau et al., 2019), e.g., people in front of buildings, cars, fences, etc.

Original: The classiﬁer-pretrained embeddings are less sensitive to these details since they are not crucial for correct classiﬁcation.
Revised: The classiﬁerpretrained embeddings are less sensitive to these details since they are not crucial for correct classiﬁcation.

Original: In contrast, self-supervised embeddings are more susceptible to small details (see Figure 2 and Table 4), hence, more images are considered “not covered”.
Revised: In contrast, self-supervised embeddings are more susceptible to small details (see Figureand Table 4), hence, more images are considered “not covered”.

Original: Figure 6 in Section C shows examples of real LSUN-Church images that are “deﬁnitely covered” by StyleGAN2 from the standpoint of InceptionV3 embeddings, but are “not covered” if SwAV embeddings are used.
Revised: Figure 6 in Section C shows examples of real LSUN-Church images that are “deﬁnitely covered” by StyleGAN2 from the standpoint of InceptionV3 embeddings, but are “not covered” if SwAV embeddings are used.

Original: More formally, we say that a real image is covered by a synthetic one with the neighborhood size k , if the distance between their embeddings does not exceed the distance from the embedding of the synthetic image to its k -th nearest neighbor in the set of all synthetic embeddings.
Revised: More formally, we say that a real image is covered by a synthetic one with the neighborhood size k , if the distance between their embeddings does not exceed the distance from the embedding of the synthetic image to its k -th nearest neighbor in the set of all synthetic embeddings.

Original: The images from Figure 6 are covered by at least 10 synthetic images with neighborhood size 5 with InceptionV3 embeddings, while being not covered even by the neighborhood of size 100 for SwAV embeddings.
Revised: The images from Figure 6 are covered by at least 10 synthetic images with neighborhood size 5 with InceptionV3 embeddings, while being not covered even by the neighborhood of size 100 for SwAV embeddings.

Original: These images possess many small details, such as monuments, cars, people, branches in the foreground, and so on, that GANs usually omit to generate. 
Revised: These images possess many small details, such as monuments, cars, people, branches in the foreground, and so on, that GANs usually omit to generate. [Figure]

Original: [Figure]
Revised: [Figure]

Original: Interestingly, the Precision values for SwAV are quite high, therefore, SwAV considers the state-of-the-art generative models being able to produce high-ﬁdelity images, but failing to generate diverse images with small details
Revised: Interestingly, the Precision values for SwAV are quite high, therefore, SwAV considers the stateof-the-art generative models being able to produce high-ﬁdelity images, but failing to generate diverse images with small details.

Original: (d) There are two signiﬁcant inconsistencies in rankings based on InceptionV3 and SwAV.
Revised: (d) There are two signiﬁcant inconsistencies in rankings based on InceptionV3 and SwAV.

Original: First, on CelebaHQ, SwAV prefers MSG* over PGGAN.
Revised: First, on CelebaHQ, SwAV prefers MSG* over PGGAN.

Original: Second, on LSUN-Bedroom, InceptionV3 ranks PGGAN higher than StyleGAN.
Revised: Second, on LSUN-Bedroom, InceptionV3 ranks PGGAN higher than StyleGAN.

Original: Since StyleGAN is a more powerful model compared to PGGAN, we also consider it as a case of  more reliable ranking with SwAV, even though it is difﬁcult to conﬁrm quantitatively due to the lack of an “oracle” measure of generation diversity.
Revised: Since StyleGAN is a more powerful model compared to PGGAN, we also consider it as a case of a more reliable ranking with SwAV, even though it is difﬁcult to conﬁrm quantitatively due to the lack of an “oracle” measure of generation diversity.

Original: 3. S AMPLE EFFICIENCY OF S W AV- BASED FID
Revised: 3. S AMPLE EFFICIENCY OF S W AV- BASED FID

Original: As an additional practical advantage of SwAV, we highlight that computing FID becomes more sample-efﬁcient compared to InceptionV3.
Revised: As an additional practical advantage of SwAV, we highlight that computing FID becomes more sample-efﬁcient compared to InceptionV3.

Original: Namely, to obtain a reliable estimation of FID values, one requires much fewer samples when using SwAV embeddings.
Revised: Namely, to obtain a reliable estimation of FID values, one requires much fewer samples when using SwAV embeddings.

Original: We illustrate this effect in Figure 4, which plots FID values w.r.t.
Revised: We illustrate this effect in Figure 4, which plots FID values w.r.t.

Original: sample size for StyleGAN2 trained on the Church dataset.
Revised: sample size for StyleGAN2 trained on Church.

Original: Since the FID values for SwAV and InceptionV3 have different typical scales, we normalize both curves by the corresponding FID value computed for a sample of size 100k.
Revised: Since the FID values for SwAV and InceptionV3 have different typical scales, we normalize both curves by the corresponding FID value computed for a sample of size 100k.

Original: FID based on SwAV embeddings converges faster, i.e., using SwAV always achieves more reliable FID estimates for a ﬁxed sample size.
Revised: FID based on SwAV embeddings converges faster, i.e., using SwAV always achieves more reliable FID estimates for a ﬁxed sample size.

Original: We attribute this beneﬁt of SwAV to the fact that its representations capture more information needed to distinguish between real and fake distributions.
Revised: We attribute this beneﬁt of SwAV to the fact that its representations capture more information needed to distinguish between real and fake distributions.

Original: Intuitively, the covariance matrices for real and synthetic data computed from SwAV embeddings are more dissimilar compared to InceptionV3based ones.
Revised: Intuitively, the covariance matrices for real and synthetic data computed from SwAV embeddings are more dissimilar compared to InceptionV3based ones.

Original: Quantitatively, the magnitude of the covariance term in FID tr ( C R + C S − 2 √ C R C S ) is larger for SwAV, which leads to smaller relative errors of its stochastic estimates.
Revised: Quantitatively, the magnitude of the covariance term in FID tr ( C R + C S − 2 √ C R C S ) is larger for SwAV, which leads to smaller relative errors of its stochastic estimates.

Original: We elaborate on this issue more rigorously in Section E.
Revised: We elaborate on this issue more rigorously in Section E.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have investigated if the state-of-the-art self-supervised models can produce more appropriate representations for GAN evaluation.
Revised: In this paper, we have investigated if the state-of-the-art self-supervised models can produce more appropriate representations for GAN evaluation.

Original: With extensive experiments, we have shown that using these representations often corrects the cases of misleading ranking obtained with classiﬁcation-pretrained embeddings.
Revised: With extensive experiments, we have shown that using these representations often corrects the cases of misleading ranking obtained with classiﬁcation-pretrained embeddings.

Original: Overall, self-supervised representations provide a more adequate GAN comparison on the four established non-Imagenet benchmarks of natural images.
Revised: Overall, self-supervised representations provide a more adequate GAN comparison on the four established non-Imagenet benchmarks of natural images.

Original: Of course, we do not claim that they should be used universally for all areas, e.g., for spectrograms or medical images.
Revised: Of course, we do not claim that they should be used universally for all areas, e.g., for spectrograms or medical images.

Original: But our work indicates that obtaining good representations needed for proper
Revised: But our work indicates that obtaining good representations needed for proper

Original: GAN evaluation does not require supervision, therefore, domain-speciﬁc self-supervised learning becomes a promising direction for further study.  
Revised: GAN evaluation does not require supervision, therefore, domain-speciﬁc self-supervised learning becomes a promising direction for further study.  A CKNOWLEDGEMENTS

Revised: We thank the Anonymous Reviewers for their reviews.

Revised: We also thank Xun Huang for commenting on his experience with SwAV on OpenReview

Original: A A PPENDIX A. T HE DETAILS OF THE VISUALIZATION TECHNIQUE IN S ECTION 3.
Revised: A A PPENDIX A. I MAGENET

Revised: [Table] A. T HE DETAILS OF THE VISUALIZATION TECHNIQUE IN S ECTION 3.

Original: The technique from Rombach et al.
Revised: The technique from Rombach et al.

Original: (2020) proposes to construct the embedding visualization in the following way.
Revised: (2020) proposes to construct the embedding visualization in the following way.

Original: Let us have an autoencoder A ( x ) , whose latent representations are denoted by z .
Revised: Let us have an autoencoder A ( x ) , whose latent representations are denoted by z .

Original: It is assumed that a latent representation z
Revised: It is assumed that a latent representation z

Original: = A ( x ) captures all the information from the image x , since the autoencoder’s goal is to fully reconstruct x from z .
Revised: = A ( x ) captures all the information from the image x , since the autoencoder’s goal is to fully reconstruct x from z .

Original: Let us denote the embedding model we want to study by E ( x ) .
Revised: Let us denote the embedding model we want to study by E ( x ) .

Original: For a particular image x , Rombach et al.
Revised: For a particular image x , Rombach et al.

Original: (2020) compute an autoencoder representation z
Revised: (2020) compute an autoencoder representation z

Original: = A ( x ) and aims to disentangle the information contained in the autoencoder representation z into the information contained in the image embedding e = E ( x ) and the information v to which the embedding model E ( x ) is invariant.
Revised: = A ( x ) and aims to disentangle the information contained in the autoencoder representation z into the information contained in the image embedding e = E ( x ) and the information v to which the embedding model E ( x ) is invariant.

Original: To this end, an invertible neural network (INN) is trained, which predicts the invariant part v = INN ( z, e ) based on the autoencoder representation and embedding.
Revised: To this end, an invertible neural network (INN) is trained, which predicts the invariant part v = INN ( z, e ) based on the autoencoder representation and embedding.

Original: Moreover, it is trained in such a way that the invariant part has a normal distribution v ∼ N (0 , 1) .
Revised: Moreover, it is trained in such a way that the invariant part has a normal distribution v ∼ N (0 , 1) .

Original: Finally, a bunch of images having the same e but different invariant parts v are produced.
Revised: Finally, a bunch of images having the same e but different invariant parts v are produced.

Original: Namely, several samples of invariant parts (cid:98) v are produced from
Revised: Namely, several samples of invariant parts (cid:98) v are produced from

Original: N (0 , 1) and the corresponding latent representations are computed from (cid:98) v and embedding e , that is (cid:98) z
Revised: N (0 , 1) and the corresponding latent representations are computed from (cid:98) v and embedding e , that is (cid:98) z

Original: = INN.reverse ( v, e ) .
Revised: = INN.reverse ( v, e ) .

Original: Then (cid:98) z can be decoded into an image using the autoencoder.
Revised: Then (cid:98) z can be decoded into an image using the autoencoder.

Original: As a result, for a given embedding model E ( x ) and a given image x , it allows to obtain a set of images with approximately the same embeddings but varying invariant part.
Revised: As a result, for a given embedding model E ( x ) and a given image x , it allows to obtain a set of images with approximately the same embeddings but varying invariant part.

Original: For the CelebaHQ dataset, we use
Revised: For the CelebaHQ dataset, we use

Original: ALAE autoencoder (Pidhorskyi et al., 2020) and train INN with the hyperparameters provided in (Rombach et al., 2020).
Revised: ALAE autoencoder (Pidhorskyi et al., 2020) and train INN with the hyperparameters provided in (Rombach et al., 2020).

Original: B C ELEBA HQ SAMPLES
Revised: B C ELEBA HQ SAMPLES

Original: [Figure]
Revised: [Figure]

Original: C C HURCH IMAGES
Revised: C C HURCH IMAGES

Original: D A UTOENCODER EMBEDDINGS
Revised: D A UTOENCODER EMBEDDINGS

Original: In this section, we compare SwAV and ALAE autoencoder (Pidhorskyi et al., 2020) embeddings.
Revised: In this section, we compare SwAV and ALAE autoencoder (Pidhorskyi et al., 2020) embeddings.

Original: ALAE was trained on Celeba (Liu et al., 2018)
Revised: ALAE was trained on Celeba (Liu et al., 2018)

Original: dataset, therefore, is expected to work correctly for the ranking of generative models on CelebaHQ.
Revised: dataset, therefore, is expected to work correctly for the ranking of generative models on CelebaHQ.

Original: To investigate what information is important for each of the embeddings, we build a dataset containing 30k real images from CelebaHQ and 30k synthetic images generated by PGGAN.
Revised: To investigate what information is important for each of the embeddings, we build a dataset containing 30k real images from CelebaHQ and 30k synthetic images generated by PGGAN.

Original: Then we select 1.5k real and 1.5k synthetic images as[Figure] queries and leave the remaining 57k pictures as a database.
Revised: Then we select 1.5k real and 1.5k synthetic images as queries and leave the remaining 57k pictures as a database.

Original: For each query, we compute three nearest neighbors from the database.
Revised: For each query, we compute three nearest neighbors from the database.

Original: Typical examples of the nearest neighbors are shown on Figure 7.
Revised: Typical examples of the nearest neighbors are shown on Figure 7.

Original: One can see that ALAE places a strong emphasis on the exact spatial arrangement, while sparsely sampled manifolds rarely include near-exact matches in terms of spatial structure (Kynk ¨ a ¨ anniemi et al., 2019).
Revised: One can see that ALAE places a strong emphasis on the exact spatial arrangement, while sparsely sampled manifolds rarely include near-exact matches in terms of spatial structure (Kynk ¨ a ¨ anniemi et al., 2019).

Original: This makes ALAE embeddings poor for GAN evaluation, for instance, via Precision and Recall metrics.
Revised: This makes ALAE embeddings poor for GAN evaluation, for instance, via Precision and Recall metrics.

Original: We also perform real/fake classiﬁcation of queries using 1-NN classiﬁcation on the constructed database.
Revised: We also perform real/fake classiﬁcation of queries using 1-NN classiﬁcation on the constructed database.

Original: The classiﬁcation accuracy is 0.729 for ALAE embeddings and 0.
Revised: The classiﬁcation accuracy is 0.729 for ALAE embeddings and 0.

Original: for SwAV.
Revised: for SwAV.

Original: Overall, the distance in the space of the autoencoder’s embeddings is less informative to distinguish between the real/fake distributions.
Revised: Overall, the distance in the space of the autoencoder’s embeddings is less informative to distinguish between the real/fake distributions.

Original: E O N THE ESTIMATE OF FID VALUES
Revised: E O N THE ESTIMATE OF FID VALUES

Original: Let us denote C R the covariance matrix for real data, C S for synthetic data and (cid:101) C R and (cid:101) C S their estimates with a ﬁnite number of samples.
Revised: Let us denote C R the covariance matrix for real data, C S for synthetic data and (cid:101) C R and (cid:101) C S their estimates with a ﬁnite number of samples.

Original: We also denote
Revised: We also denote

Original: [Equation]
Revised: [Equation]

Original: In (Dowson & Landau, 1982) it was shown that d ( C R , C S ) deﬁnes a metric on the space of all covariance matrices of order n .
Revised: In (Dowson & Landau, 1982) it was shown that d ( C R , C S ) deﬁnes a metric on the space of all covariance matrices of order n .

Original: Our goal is to assess the relative error of the FID estimate.
Revised: Our goal is to assess the relative error of the FID estimate.

Original: Under the assumption that the means of real and synthetic data distributions are estimated quite accurately (which is the case already for a few thousand of samples) we need to estimate only 
Revised: Under the assumption that the means of real and synthetic data distributions are estimated quite accurately (which is the case already for a few thousand of samples) we need to estimate only [Equation] Query

Original: [Equation]
Revised: [Equation]

Revised: ALAE

Original: Due to the metric properties
Revised: Due to the metric properties

Original: [Equation]
Revised: [Equation]

Original: Then 
Revised: Then d

Original: [Equation]
Revised: [Equation]

Original: Due to the symmetry the same inequality can be obtained with the opposite sign and as the result we get
Revised: Due to the symmetry the same inequality can be obtained with the opposite sign and as the result we get

Original: [Equation]
Revised: [Equation]

Original: where the numerator corresponds to the accuracy of the estimation of covariance matrices and the denominator corresponds to the distance between covariance matrices for real and synthetic data.
Revised: where the numerator corresponds to the accuracy of the estimation of covariance matrices and the denominator corresponds to the distance between covariance matrices for real and synthetic data.

Original: Thus, larger values of d ( C R , C S ) result in a smaller relative error of the FID estimation.
Revised: Thus, larger values of d ( C R , C S ) result in a smaller relative error of the FID estimation.

Original: Experimentally, we compute the distances d ( C R , C S ) =
Revised: Experimentally, we compute the distances d ( C R , C S ) =

Original: tr ( C R + C S − 2 √ C R C S ) based on SwAV and
Revised: tr ( C R + C S − 2 √ C R C S ) based on SwAV and

Original: InceptionV3 embeddings for StyleGAN2 trained on the Church dataset.
Revised: InceptionV3 embeddings for StyleGAN2 trained on the Church dataset.

Original: 1 We obtain 0 .
Revised: 3 We obtain 0 .

Original: 083 for SwAV and0 .
Revised: SwAV and

Original:  for InceptionV3, which conﬁrms the validity of calculations above and explains the better sample-efﬁciency of SwAV presented in Figure 4. 
Revised: 028 for InceptionV3, which conﬁrms the validity of calculations above and explains the better sample-efﬁciency of SwAV presented in Figure 4. [Table]

Revised: F H UMAN EVALUATION

Revised: As an additional evidence that inter-image distances induced by SwAV are better aligned with human perception compared to InceptionV

Revised: [Table]

Revised: we perform two crowdsourcing experiments.

Original: F
Revised: All the data and labellings are released on the GitHub. 4 . SwAV-based Precision and Recall have higher agreement with human judgements. The key step of computing Recall is checking, if for a given real embedding r there exists a generated embedding g that is closer to r than its k -th real neighbor. To verify, if a particular embedding agrees well with human perception, we perform the following procedure.

Revised: We form a triplet of an anchor real image I anchor that contributes to the Recall value, its 5 -th closest neighbor among the real images I 5th and the generated image I gen that appears to be closer to I anchor in terms of the considered embedding.

Revised: A human assessor is then asked to choose an image between I gen and I 5th that is more similar to I anchor .

Revised: Once the assessor chooses the generated one, we consider it as a case of agreement with the embedding.

Revised: The embeddings with higher agreement rate are more suitable for computing Recall.

Revised: For Precision, we similarly form the triplets consisting of a generated image, its 5 -th neighbor among the generated images and a real image I real closer to it I anchor , I 5th , I real .

Revised: Once an assessor answers that I real is more similar to I anchor then I 5th , we consider this is as an agreement with the embedding.

Revised: Here we always use the same real and generated samples as for the evaluation of the metrics in

Revised: Section 3.

Revised: We label three datasets with two GAN models for each.

Revised: For each pair of a dataset and a generator, we label 200 different triplets, each by ten different assessors.

Revised: An assessor is also able to choose the options “equally similar” or “both completely dissimilar”.

Revised: Once the “equally similar” is chosen, we suppose that the agreement happens with the probability 0 .

Revised: The user interface is illustrated on Figure 8 (left).

Revised: All the labeling was performed in Yandex Toloka 5 .

Revised: The results are presented on Table 7 and conﬁrm that SwAV emebddings mostly have higher agreement with human perception.

Revised: Quality of neighbors.

Revised: As a more simple experiment, we also ask human assessors to compare quality of top- 5 neighbors produced by InceptionV3 and SwAV embeddings.

Revised: Namely, we take a set of N real images I , same as in Section 3.

Revised: For a given real image r ∈

Revised: I we form two lists of its 5 nearest neighbors B IV3 ⊂ I and B SwAV ⊂ I based on InceptionV3 and SwAV embeddings.

Revised: An assessor is asked to assign r either to B IV3 or to B SwAV .

Revised: Same as above, the assessor may also label it as “equal” which is treated as an equal probability of each set to be chosen.

Revised: The user interface is illustrated on Figure 8 (right).

Revised: For each dataset we form 500 different triplets r, B SwAV , B IV3 , each labeled by ten different assessors.

Revised: Table 8 presents probabilities that assessors prefer a SwAV- based set of neighbors, indicating that SwAV-induced distances better capture perceptual similarity compared to InceptionV3.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: com/

Revised: [Table]

Revised: [Table]

Revised: https://

Revised: [Table]

Revised: com/

Revised: [Table]

Revised: [Table]

Revised: https://

Revised: [Table]

Revised: com/hubert0527/COCO

Revised: [Table]

Revised: GAN/tree/ 12b90e26e23214c2072c9701644e9724e052743c

Revised: [Table]

Revised: _N2M2S

Revised: [Table]

Revised: [Table]

Revised: .com/

Revised: NVlabs

Revised: [Table]

Revised: //github.com/NVlabs/stylegan

Original: T
Revised: [Table] [Table] [Table] .com/tkarras

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: com/

Revised: [Table]

Revised: stylegan

Revised: [Table]

Revised: [Table]

Revised: com/

Revised: [Table]

Revised: [Table]

Revised: stylegan

Revised: [Table]

Revised: [Table]

Original: Representation
Revised: Representation

Original: Dim
Revised: Dim

Original: URL
Revised: URL

Original: Checkpoint
Revised: Checkpoint

Original: InceptionV 2048 https://github.
Revised: InceptionV 2048 https://github.

Original: com/mseitzer/ pytorch-fid https://github.com/ mseitzer/pytorch-fid/ releases/download/fid_ weights/pt_inception-201512-05-6726825d.pth
Revised: com/mseitzer/ pytorch-fid https://github.com/ mseitzer/pytorch-fid/ releases/download/fid_ weights/pt_inception-201512-05-6726825d.pth

Original: Resnet2048 https://pytorch.
Revised: Resnet2048 https://pytorch.

Original: org/docs/stable/ torchvision/
Revised: org/docs/stable/ torchvision/

Original: models.html https://download.pytorch.
Revised: models.html https://download.pytorch.

Original: org/models/resnet5019c8e357.pth
Revised: org/models/resnet5019c8e357.pth

Original: Imagenet21k 2048 https://tfhub.
Revised: Imagenet21k 2048 https://tfhub.

Original: dev/google/bit/mr50x1/ https://tfhub.dev/google/ bit/m-r50x1/
Revised: dev/google/bit/mr50x1/ https://tfhub.dev/google/ bit/m-r50x1/

Original: SwAV 2048 https:// github.com/ facebookresearch/ swav https://dl.fbaipublicfiles.
Revised: SwAV 2048 https:// github.com/ facebookresearch/ swav https://dl.fbaipublicfiles.

Original: com/deepcluster/swav_800ep_ pretrain.pth.tar
Revised: com/deepcluster/swav_800ep_ pretrain.pth.tar

Original: DeepClusterV2 2048 https:// github.com/ facebookresearch/ swav https://dl.fbaipublicfiles.
Revised: DeepClusterV2 2048 https:// github.com/ facebookresearch/ swav https://dl.fbaipublicfiles.

Original: com/deepcluster/ deepclusterv2_800ep_ pretrain.pth.tar
Revised: com/deepcluster/ deepclusterv2_800ep_ pretrain.pth.tar

Original: MoCoV 2048 https:// github.com/ facebookresearch/ moco https://dl.fbaipublicfiles.
Revised: MoCoV 2048 https:// github.com/ facebookresearch/ moco https://dl.fbaipublicfiles.

Original: com/moco/moco_checkpoints/ moco_v2_800ep/moco_v2_ 800ep_pretrain.pth.tar
Revised: com/moco/moco_checkpoints/ moco_v2_800ep/moco_v2_ 800ep_pretrain.pth.tar