Original: R ARITY S CORE : A N EW M ETRIC TO E VALUATE THE U NCOMMONNESS OF S YNTHESIZED I MAGES
Revised: R ARITY S CORE : A N EW M ETRIC TO E VALUATE THE U NCOMMONNESS OF S YNTHESIZED I MAGES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Evaluation metrics in image synthesis play a key role to measure performances of generative models.
Revised: Evaluation metrics in image synthesis play a key role to measure performances of generative models.

Original: However, most metrics mainly focus on image fidelity.
Revised: However, most metrics mainly focus on image fidelity.

Original: Existing diversity metrics are derived by comparing distributions, and thus they cannot quantify the diversity or rarity degree of each generated image.
Revised: Existing diversity metrics are derived by comparing distributions, and thus they cannot quantify the diversity or rarity degree of each generated image.

Original: In this work, we propose a new evaluation metric, called ‘rarity score’, to measure both imagewise uncommonness and model-wise diversified generation performance.
Revised: In this work, we propose a new evaluation metric, called ‘rarity score’, to measure both imagewise uncommonness and model-wise diversified generation performance.

Original: We first show empirical observation that common samples are close to each other and rare samples are far from each other in nearest-neighbor distances on latent spaces represented by feature extractor networks such as VGG16.
Revised: We first show empirical observation that common samples are close to each other and rare samples are far from each other in nearest-neighbor distances on latent spaces represented by feature extractor networks such as VGG16.

Original: We then use our metric to demonstrate that the extent to which different generative models produce rare images can be effectively compared.
Revised: We then use our metric to demonstrate that the extent to which different generative models produce rare images can be effectively compared.

Original: We also propose a method to compare rarities between datasets that share the same concept such as CelebA-HQ and FFHQ.
Revised: We also propose a method to compare rarities between datasets that share the same concept such as CelebA-HQ and FFHQ.

Original: Finally, we analyze the use of metrics in different designs of feature extractors to better understand the relationship between feature spaces and resulting high-rarity images.
Revised: Finally, we analyze the use of metrics in different designs of feature extractors to better understand the relationship between feature spaces and resulting high-rarity images.

Original: Code will be publicly available for the research community.
Revised: Code will be publicly available for the research community.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative models have attracted a lot of attention for recent years.
Revised: Generative models have attracted a lot of attention for recent years.

Original: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have achieved significant advancement over the past several years, enabling many computer vision tasks such as image manipulation (Kim et al., 2021;
Revised: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have achieved significant advancement over the past several years, enabling many computer vision tasks such as image manipulation (Kim et al., 2021;

Original: Jahanian et al., 2020;
Revised: Jahanian et al., 2020;

Original: Shen et al., 2020;
Revised: Shen et al., 2020;

Original: H ¨ ark ¨ onen et al., 2020;
Revised: H ¨ ark ¨ onen et al., 2020;

Original: Bau et al., 2019), domain translation (Isola et al., 2017; Zhu et al., 2017; Choi et al., 2018; 2020;
Revised: Bau et al., 2019), domain translation (Isola et al., 2017; Zhu et al., 2017; Choi et al., 2018; 2020;

Original: Kim et al., 2020; 2019), and image or video generation (Kim et al., 2022; Kim & Ha, 2022; Lee et al., 2022; Karras et al., 2019; 2020b;a; 2021;
Revised: Kim et al., 2020; 2019), and image or video generation (Kim et al., 2022; Kim & Ha, 2022; Lee et al., 2022; Karras et al., 2019; 2020b;a; 2021;

Original: Skorokhodov et al., 2022; Yu et al., 2022b; Tian et al., 2021; Tulyakov et al., 2018).
Revised: Skorokhodov et al., 2022; Yu et al., 2022b; Tian et al., 2021; Tulyakov et al., 2018).

Original: The emergence of diffusion models accelerates the advancements of the generative models especially in the field of text-to-image modeling (Ramesh et al., 2022; Saharia et al., 2022; Yu et al., 2022a; Rombach et al., 2022).
Revised: The emergence of diffusion models accelerates the advancements of the generative models especially in the field of text-to-image modeling (Ramesh et al., 2022; Saharia et al., 2022; Yu et al., 2022a; Rombach et al., 2022).

Original: To quantify the performance of generative models, various metrics have been proposed.
Revised: To quantify the performance of generative models, various metrics have been proposed.

Original: As standard evaluation metrics, inception score (IS) (Salimans et al., 2016), kernel inception distance (KID) (Bi´nkowski et al., 2018), and Frech´et inception distance (FID) (Heusel et al., 2017) are prevalent for evaluating the quality of images synthesized by generative models.
Revised: As standard evaluation metrics, inception score (IS) (Salimans et al., 2016), kernel inception distance (KID) (Bi´nkowski et al., 2018), and Frech´et inception distance (FID) (Heusel et al., 2017) are prevalent for evaluating the quality of images synthesized by generative models.

Original: These metrics evaluate the discrepancy between generated and real image sets on the feature space characterized by a generative model with respect to diversity and fidelity.
Revised: These metrics evaluate the discrepancy between generated and real image sets on the feature space characterized by a generative model with respect to diversity and fidelity.

Original: The fidelity represents the quality of the generated image, and the diversity indicates that the generator creates images without mode collapse, similar to the distribution of the training datasets.
Revised: The fidelity represents the quality of the generated image, and the diversity indicates that the generator creates images without mode collapse, similar to the distribution of the training datasets.

Original: To improve the fidelity and diversity, it is required that the distribution of generated images is similar to the real image distribution (Kynk¨a¨anniemi et al., 2019; Naeem et al., 2020).
Revised: To improve the fidelity and diversity, it is required that the distribution of generated images is similar to the real image distribution (Kynk¨a¨anniemi et al., 2019; Naeem et al., 2020).

Original: Nevertheless, reproducing capability of the low density area is often overlooked.
Revised: Nevertheless, reproducing capability of the low density area is often overlooked.

Original: If we consider the distribution of training data, it is clear that the rare samples in the low density areas are poorly represented and thus likely to be difficult for the generator to reproduce.
Revised: If we consider the distribution of training data, it is clear that the rare samples in the low density areas are poorly represented and thus likely to be difficult for the generator to reproduce.

Original: Examples of rare samples from public datasets include people with various accessories in FFHQ (Karras et al., 2019), white animals in AFHQ (Choi et al., 2020), and uncommon statues in MetFaces (Karras et al., 2020a).
Revised: Examples of rare samples from public datasets include people with various accessories in FFHQ (Karras et al., 2019), white animals in AFHQ (Choi et al., 2020), and uncommon statues in MetFaces (Karras et al., 2020a).

Original: Assessing the rarity is important not only because it is related to assessing the reproducing capability of the generative models, but also because it is related to selecting generated images.
Revised: Assessing the rarity is important not only because it is related to assessing the reproducing capability of the generative models, but also because it is related to selecting generated images.

Original: Recent
Revised: Recent

Original: [Figure]
Revised: [Figure]

Original: A F HQ -
Revised: A F HQ -

Original: [Figure]
Revised: [Figure]

Original: il d
Revised: il d

Original: [Figure]
Revised: [Figure]

Original: t F
Revised: t F

Original: [Figure]
Revised: [Figure]

Original: commercialization of text-to-image models such as DALL-E 2 (Ramesh et al., 2022) and Stable Diffusion (Rombach et al., 2022) enhance the necessity of metrics for uncommonness and creativity.
Revised: commercialization of text-to-image models such as DALL-E 2 (Ramesh et al., 2022) and Stable Diffusion (Rombach et al., 2022) enhance the necessity of metrics for uncommonness and creativity.

Original: In these creative AI application scenarios, a metric that measures each synthesized image (instancewise metric) is essential for users and consumers to select images among the ones provided through
Revised: In these creative AI application scenarios, a metric that measures each synthesized image (instancewise metric) is essential for users and consumers to select images among the ones provided through

Original: API, rather than model-wise metrics such as FID (Heusel et al., 2017) and LPIPS (Zhang et al., 2018).
Revised: API, rather than model-wise metrics such as FID (Heusel et al., 2017) and LPIPS (Zhang et al., 2018).

Original: Unfortunately, no instance-wise metric exists for measuring the creative and uncommon degree of each image despite its practicality and necessity.
Revised: Unfortunately, no instance-wise metric exists for measuring the creative and uncommon degree of each image despite its practicality and necessity.

Original: In this paper, we propose a novel generative model evaluation metric that can represent the generative capabilities of rare samples in generative models as scores (a.k.a.
Revised: In this paper, we propose a novel generative model evaluation metric that can represent the generative capabilities of rare samples in generative models as scores (a.k.a.

Original: rarity score).
Revised: rarity score).

Original: Our metric contributes to classifying rare images and typical images similar to those frequently observed in training datasets.
Revised: Our metric contributes to classifying rare images and typical images similar to those frequently observed in training datasets.

Original: The proposed rarity metric highlights the open problem of sparse generation of rare samples from generative models.
Revised: The proposed rarity metric highlights the open problem of sparse generation of rare samples from generative models.

Original: Additionally, we have conducted comparative experiments on which of the previous state-of-the-art models produce more rare samples with preserving quality performances.
Revised: Additionally, we have conducted comparative experiments on which of the previous state-of-the-art models produce more rare samples with preserving quality performances.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We propose the first metric to quantify the rarity of individual generation which existing metrics cannot provide.
Revised: • We propose the first metric to quantify the rarity of individual generation which existing metrics cannot provide.

Original: Using the proposed metric, generations with the desired degree of rarity can be sampled.
Revised: Using the proposed metric, generations with the desired degree of rarity can be sampled.

Original: • We show that the proposed metric can be used to compare the ability of models to generate rare samples.
Revised: • We show that the proposed metric can be used to compare the ability of models to generate rare samples.

Original: The proposed metric can further be used to evaluate which dataset contains more rare samples among the datasets that share the same concept such as CelebA-HQ and FFHQ datasets.
Revised: The proposed metric can further be used to evaluate which dataset contains more rare samples among the datasets that share the same concept such as CelebA-HQ and FFHQ datasets.

Original: • We show the proposed metric can be applied on top of the various feature spaces with different viewpoints
Revised: • We show the proposed metric can be applied on top of the various feature spaces with different viewpoints

Original: of rarity,
Revised: of rarity,

Original: by analyzing the sampled generations.
Revised: by analyzing the sampled generations.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Precision and Recall Precision and recall are commonly used performance metrics in many areas including classification tasks or natural language processing.
Revised: Precision and Recall Precision and recall are commonly used performance metrics in many areas including classification tasks or natural language processing.

Original: In specific, to quantify the performance of the generative models, precision measures the fraction of the fake distribution that can be generated from the real distribution.
Revised: In specific, to quantify the performance of the generative models, precision measures the fraction of the fake distribution that can be generated from the real distribution.

Original: On the other hand, recall measures the fraction of the true distribution which can be reproduced by the fake distribution.
Revised: On the other hand, recall measures the fraction of the true distribution which can be reproduced by the fake distribution.

Original: Practically, k -nearest neighbor ( kNN) based method has been proposed to estimate the real and fake manifolds (Kynk¨a¨anniemi et al., 2019).
Revised: Practically, k -nearest neighbor ( kNN) based method has been proposed to estimate the real and fake manifolds (Kynk¨a¨anniemi et al., 2019).

Original: For real samples X r ∼ P r and fake samples X g ∼ P g , we first embed them in the highdimensional feature space using pretrained DNNs such as VGG16 (andAndrew Zisserman, 2015) or the image encoder of CLIP (Radford et al., 2021) to get sets of feature vectors Φ r and Φ g , respectively.
Revised: For real samples X r ∼ P r and fake samples X g ∼ P g , we first embed them in the highdimensional feature space using pretrained DNNs such as VGG16 (andAndrew Zisserman, 2015) or the image encoder of CLIP (Radford et al., 2021) to get sets of feature vectors Φ r and Φ g , respectively.

Original: The real and fake manifolds are estimated by the sets of k -NN spheres of each sample as follows.
Revised: The real and fake manifolds are estimated by the sets of k -NN spheres of each sample as follows.

Original: [Equation]
Revised: [Equation]

Original: Here, NN k ( ϕ i , Φ ) represents the distance between ϕ i and its k -th nearest neighbor in Φ .
Revised: Here, NN k ( ϕ i , Φ ) represents the distance between ϕ i and its k -th nearest neighbor in Φ .

Original: B k
Revised: B k

Original: ( ϕ i , Φ ) is the k -NN sphere of ϕ i with the radius of NN k ( ϕ i , Φ ) defined as a set of all ϕ whose distance to ϕ i is smaller than or equal to NN k
Revised: ( ϕ i , Φ ) is the k -NN sphere of ϕ i with the radius of NN k ( ϕ i , Φ ) defined as a set of all ϕ whose distance to ϕ i is smaller than or equal to NN k

Original: ( ϕ i , Φ ) .
Revised: ( ϕ i , Φ ) .

Original: For the distance metric d , we use L2 distance for the rest of the paper.
Revised: For the distance metric d , we use L2 distance for the rest of the paper.

Original: Then, precision and recall are respectively defined as
Revised: Then, precision and recall are respectively defined as

Original: [Equation]
Revised: [Equation]

Original: where I is indicator function.
Revised: where I is indicator function.

Original: Precision often maps with the fidelity and recall maps to the diversity of the generations.
Revised: Precision often maps with the fidelity and recall maps to the diversity of the generations.

Original: However, it is nontrivial to apply both precision and recall to an individual generation as they are designed to measure on the sets of samples.
Revised: However, it is nontrivial to apply both precision and recall to an individual generation as they are designed to measure on the sets of samples.

Original: To quantify the image quality of an individual generation, realism score has been proposed.
Revised: To quantify the image quality of an individual generation, realism score has been proposed.

Original: realism score measures the maximum of the inverse relative distance of a fake sample in a real k -NN sphere.
Revised: realism score measures the maximum of the inverse relative distance of a fake sample in a real k -NN sphere.

Original: [Equation]
Revised: [Equation]

Original: Realism score is high when the distance between the given fake sample and a real sample is relatively small compared to the radius of the k -NN sphere of the real sample.
Revised: Realism score is high when the distance between the given fake sample and a real sample is relatively small compared to the radius of the k -NN sphere of the real sample.

Original: However, the realism score is not suitable to find rare samples as realism score can be large even if the radius of the k -NN sphere is small when the distance between fake sample and a real sample is also very small.
Revised: However, the realism score is not suitable to find rare samples as realism score can be large even if the radius of the k -NN sphere is small when the distance between fake sample and a real sample is also very small.

Original: Density and Coverage While precision and recall are commonly used, there are still several drawbacks that they are vulnerable to the outliers and computationally inefficient.
Revised: Density and Coverage While precision and recall are commonly used, there are still several drawbacks that they are vulnerable to the outliers and computationally inefficient.

Original: To overcome the drawbacks, density and coverage have been proposed (Naeem et al., 2020).
Revised: To overcome the drawbacks, density and coverage have been proposed (Naeem et al., 2020).

Original: While precision only counts whether the fake sample is inside the real manifold or not, density counts the number of the real k -NN spheres that contains the fake sample.
Revised: While precision only counts whether the fake sample is inside the real manifold or not, density counts the number of the real k -NN spheres that contains the fake sample.

Original: [Equation]
Revised: [Equation]

Original: If a fake sample is included in the multiple real k -NN spheres, it is more certain that the fake sample is inside the real manifold.
Revised: If a fake sample is included in the multiple real k -NN spheres, it is more certain that the fake sample is inside the real manifold.

Original: Thus, density can be more robust to the outliers compared to the precision.
Revised: Thus, density can be more robust to the outliers compared to the precision.

Original: On the other hand, similar to recall, coverage is relevant to the diversity of the generations.
Revised: On the other hand, similar to recall, coverage is relevant to the diversity of the generations.

Original: Coverage computes the number of real samples containing at least one fake sample in its k -NN sphere.
Revised: Coverage computes the number of real samples containing at least one fake sample in its k -NN sphere.

Original: [Equation]
Revised: [Equation]

Original: If a fake sample is sparsely located, the fake manifold can be exaggerated and the recall can be overestimated.
Revised: If a fake sample is sparsely located, the fake manifold can be exaggerated and the recall can be overestimated.

Original: As real manifold is known to have less outliers than the fake manifold, coverage can prevent such overestimation.
Revised: As real manifold is known to have less outliers than the fake manifold, coverage can prevent such overestimation.

Original: Furthermore, coverage is more computationally efficient compared to the recall as coverage does not require the k -NN computations of fake samples.
Revised: Furthermore, coverage is more computationally efficient compared to the recall as coverage does not require the k -NN computations of fake samples.

Original: However, density and coverage are also not suitable to be applied on individual generations as they work on a set of generations.
Revised: However, density and coverage are also not suitable to be applied on individual generations as they work on a set of generations.

Original: Truncation Trick
Revised: Truncation Trick

Original: The truncation trick is widely used in generative models to restrict latent sampling space to maintain the image quality (Kingma & Dhariwal, 2018; Brock et al., 2019; Karras et al., 2019).
Revised: The truncation trick is widely used in generative models to restrict latent sampling space to maintain the image quality (Kingma & Dhariwal, 2018; Brock et al., 2019; Karras et al., 2019).

Original: Typically, in StyleGAN based models (Karras et al., 2019; 2020b;a), the truncation trick is applied in the warped latent space W from the input latent space Z with f : Z → W as follows.
Revised: Typically, in StyleGAN based models (Karras et al., 2019; 2020b;a), the truncation trick is applied in the warped latent space W from the input latent space Z with f : Z → W as follows.

Original: [Equation]
Revised: [Equation]

Original: When ψ = 1 , it is the same as not using the truncation trick.
Revised: When ψ = 1 , it is the same as not using the truncation trick.

Original: As ψ decreases, the latent code shifts toward the mean latent code.
Revised: As ψ decreases, the latent code shifts toward the mean latent code.

Original: When ψ = 0 , w ′ becomes ¯w , which is the mean latent code in W .
Revised: When ψ = 0 , w ′ becomes ¯w , which is the mean latent code in W .

Original: Truncation trick is known to increase the fidelity at the expense of lowering the diversity of the generations.
Revised: Truncation trick is known to increase the fidelity at the expense of lowering the diversity of the generations.

Original: Real
Revised: Real

Original: Fake 𝑟𝑔𝑑𝑟𝑔𝑟𝑑 2 𝑑𝑔
Revised: Fake 𝑟𝑔𝑑𝑟𝑔𝑟𝑑 2 𝑑𝑔

Original: [Figure]
Revised: [Figure]

Original:  R ARITY S CORE
Revised:  R ARITY S CORE

Original: To evaluate the rarity of both individual generated instance and generative model performance, we propose a novel metric called rarity score .
Revised: To evaluate the rarity of both individual generated instance and generative model performance, we propose a novel metric called rarity score .

Original: Following the idea of Kynk¨a¨anniemi et al.
Revised: Following the idea of Kynk¨a¨anniemi et al.

Original: and Naeem et al.
Revised: and Naeem et al.

Original: (2020), we use k -NN to represent the manifolds of real and generated samples as in Figure 2.
Revised: (2020), we use k -NN to represent the manifolds of real and generated samples as in Figure 2.

Original: 3. I NSTANCE - WISE R ARITY S CORE
Revised: 3. I NSTANCE - WISE R ARITY S CORE

Original: We hypothesize that ordinary samples would be closer to each other whereas unique and rare samples would be sparsely located in the feature space.
Revised: We hypothesize that ordinary samples would be closer to each other whereas unique and rare samples would be sparsely located in the feature space.

Original: Figure 1 shows the real samples with both the smallest nearest neighbor distances (NNDs) and the largest NNDs from the training dataset.
Revised: Figure 1 shows the real samples with both the smallest nearest neighbor distances (NNDs) and the largest NNDs from the training dataset.

Original: For all datasets, samples with the smallest NNDs show representative and typical images.
Revised: For all datasets, samples with the smallest NNDs show representative and typical images.

Original: On the contrary, the samples with the largest NNDs have strong individuality and are significantly different from the typical images with the smallest NNDs.
Revised: On the contrary, the samples with the largest NNDs have strong individuality and are significantly different from the typical images with the smallest NNDs.

Original: From this intuition, we propose the rarity score:
Revised: From this intuition, we propose the rarity score:

Original: [Equation]
Revised: [Equation]

Original: Equation (8) measures the rarity score of the given fake sample as the minimum radius of k -NN sphere of the real sample that contains the given fake sample.
Revised: Equation (8) measures the rarity score of the given fake sample as the minimum radius of k -NN sphere of the real sample that contains the given fake sample.

Original: Equation (8) can be viewed as a dual-way k-NN, if we see the constraint ϕ g ∈ B k ( ϕ r , Φ r ) as a reverse k-NN (RkNN) (Korn &
Revised: Equation (8) can be viewed as a dual-way k-NN, if we see the constraint ϕ g ∈ B k ( ϕ r , Φ r ) as a reverse k-NN (RkNN) (Korn &

Original: Muthukrishnan, 2000; Tao et al., 2004).
Revised: Muthukrishnan, 2000; Tao et al., 2004).

Original: RkNN is known as a problem to find the set of samples to which the given query is at least the k-th most influential.
Revised: RkNN is known as a problem to find the set of samples to which the given query is at least the k-th most influential.

Original: Thus we are filtering real samples to those that can be influential to the given fake sample.
Revised: Thus we are filtering real samples to those that can be influential to the given fake sample.

Original: Equation (8) then takes 1-NN to the RkNN set.
Revised: Equation (8) then takes 1-NN to the RkNN set.

Original: In Case 1 in Figure 2, g 1 is in the k -NN sphere of r 1 , thus the rarity score of g 1 is the radius of the k -NN sphere of r 1 , which is d 1 .
Revised: In Case 1 in Figure 2, g 1 is in the k -NN sphere of r 1 , thus the rarity score of g 1 is the radius of the k -NN sphere of r 1 , which is d 1 .

Original: As samples outside of the real manifold cannot guarantee their high fidelity, our metric does not consider the generations not contained in the real manifold manifold k ( Φ r ) .
Revised: As samples outside of the real manifold cannot guarantee their high fidelity, our metric does not consider the generations not contained in the real manifold manifold k ( Φ r ) .

Original: For example, the generation g 3 in Figure 2 is an artifact and it is out of the real manifold.
Revised: For example, the generation g 3 in Figure 2 is an artifact and it is out of the real manifold.

Original: Thus the rarity score for g 3 is undefined.
Revised: Thus the rarity score for g 3 is undefined.

Original: This ensures the fidelity of the selected images without any additional fidelity metrics.
Revised: This ensures the fidelity of the selected images without any additional fidelity metrics.

Original: The synthesized examples out of the real manifold are presented in Appendix E.
Revised: The synthesized examples out of the real manifold are presented in Appendix E.

Original: Choice of k
Revised: Choice of k

Original: We show how k for nearest neighbors affects the proposed metric in terms of the
Revised: We show how k for nearest neighbors affects the proposed metric in terms of the

Original: Spearman rank correlation.
Revised: Spearman rank correlation.

Original: Figure 3 shows the rank correlations among 1 ≤ k ≤ 10 .
Revised: Figure 3 shows the rank correlations among 1 ≤ k ≤ 10 .

Original: The left heatmap shows the rank correlation matrix for the samples included in M 1 where M i indicates manifold i ( Φ r ) , i.e., the real manifold built with i -NN spheres.
Revised: The left heatmap shows the rank correlation matrix for the samples included in M 1 where M i indicates manifold i ( Φ r ) , i.e., the real manifold built with i -NN spheres.

Original: The second plot in Figure 3 shows the average rank correlation between k = 1 and 1 ≤ k ′ ≤ 10 .
Revised: The second plot in Figure 3 shows the average rank correlation between k = 1 and 1 ≤ k ′ ≤ 10 .

Original: Each color indicates the target generations in M i .
Revised: Each color indicates the target generations in M i .

Original: Please note that M i ⊂ M j for i ≤ j and the rank correlation changes dramatically for the cases in M j \ M i .
Revised: Please note that M i ⊂ M j for i ≤ j and the rank correlation changes dramatically for the cases in M j \ M i .

Original: For example, for M 3 , the mean rank correlations have high values ≥ 0 .
Revised: For example, for M 3 , the mean rank correlations have high values ≥ 0 .

Original: 8 for all k ≥ 3 .
Revised: 8 for all k ≥ 3 .

Original: While there exists slight drop in the mean rank correlation as i increases, due to the change in the out-of-manifold (OOM) samples, we can see that the mean rank correlation is consistent for k ≥ i .
Revised: While there exists slight drop in the mean rank correlation as i increases, due to the change in the out-of-manifold (OOM) samples, we can see that the mean rank correlation is consistent for k ≥ i .

Original: In other words, the rank of the proposed metric does not change much for different choices of k , and we can say the proposed metric is robust to the choice of k .
Revised: In other words, the rank of the proposed metric does not change much for different choices of k , and we can say the proposed metric is robust to the choice of k .

Original: However, if we decrease k , the number of fake samples out of the real manifold increases as we can see in the graph on the right in Figure 3.
Revised: However, if we decrease k , the number of fake samples out of the real manifold increases as we can see in the graph on the right in Figure 3.

Original: Considering that k ≥ 3 keeps OOM samples under 30%, we set k = 3 for the rest of this paper.
Revised: Considering that k ≥ 3 keeps OOM samples under 30%, we set k = 3 for the rest of this paper.

Original: Further analyses of false-positive versus false-negative-rate in Appendix G support the choice of k = 3 .
Revised: Further analyses of false-positive versus false-negative-rate in Appendix G support the choice of k = 3 .

Original: 3. M ODEL - WISE R ARITY S CORE
Revised: 3. M ODEL - WISE R ARITY S CORE

Original: Model-wise Measurement As rarity score is defined as an instance-wise measurement, there can be many ways to aggregate the rarity scores of a set of fake samples to compare between generative models.
Revised: Model-wise Measurement As rarity score is defined as an instance-wise measurement, there can be many ways to aggregate the rarity scores of a set of fake samples to compare between generative models.

Original: In this paper, we suggest one way of model-wise measurement, RS-p , based on the rarity score.
Revised: In this paper, we suggest one way of model-wise measurement, RS-p , based on the rarity score.

Original: [Equation]
Revised: [Equation]

Original: where Φ Θ g denotes the set of fake samples generated by a model Θ .
Revised: where Φ Θ g denotes the set of fake samples generated by a model Θ .

Original: We omit Φ r in Rarity as it is fixed.
Revised: We omit Φ r in Rarity as it is fixed.

Original: Basically, RS-p is the mean rarity score of the fake samples with the top p% rarity scores.
Revised: Basically, RS-p is the mean rarity score of the fake samples with the top p% rarity scores.

Original: RS-p is designed to compare the rarity of the most rare generations.
Revised: RS-p is designed to compare the rarity of the most rare generations.

Original: Relationship with Existing Metrics Here, we mainly investigate the relationship between our rarity and two existing methods that are most relevant to ours: the realism score (Kynk ¨ a ¨ anniemi et al., 2019) and the truncation trick (Brock et al., 2019).
Revised: Relationship with Existing Metrics Here, we mainly investigate the relationship between our rarity and two existing methods that are most relevant to ours: the realism score (Kynk ¨ a ¨ anniemi et al., 2019) and the truncation trick (Brock et al., 2019).

Original: The realism score is a metric which measures the fidelity of an individual generation.
Revised: The realism score is a metric which measures the fidelity of an individual generation.

Original: While the realism score takes into account the radius of the real k -NN sphere, the effect of the radius is degraded by the effect of the distance between the real and the fake sample.
Revised: While the realism score takes into account the radius of the real k -NN sphere, the effect of the radius is degraded by the effect of the distance between the real and the fake sample.

Original: Realism score can be high for the samples in the high density region which is assumed to be typical and thus not suitable to measure rarity.
Revised: Realism score can be high for the samples in the high density region which is assumed to be typical and thus not suitable to measure rarity.

Original: In Figure 4, while images with the high realism scores show high fidelity, the images with the high rarity scores show more extraordinary characteristics in the images.
Revised: In Figure 4, while images with the high realism scores show high fidelity, the images with the high rarity scores show more extraordinary characteristics in the images.

Original: Truncation trick has been proposed to maintain high fidelity of generations by filtering out the outskirt latent codes far from the mean latent code.
Revised: Truncation trick has been proposed to maintain high fidelity of generations by filtering out the outskirt latent codes far from the mean latent code.

Original: This method can increase the fidelity of the images, but it can compromise its versatility.
Revised: This method can increase the fidelity of the images, but it can compromise its versatility.

Original: If we use ψ ≥ 1 to shift the latent code in the opposite direction from the mean latent code, we might get more diverse images.
Revised: If we use ψ ≥ 1 to shift the latent code in the opposite direction from the mean latent code, we might get more diverse images.

Original: In this case, however, we cannot guarantee the fidelity of images as shown in Figure 4.
Revised: In this case, however, we cannot guarantee the fidelity of images as shown in Figure 4.

Original: As it is known that the truncation trick shows diversity-fidelity trade-off, it will be convenient to compare with other metrics based on the truncation parameter changes.
Revised: As it is known that the truncation trick shows diversity-fidelity trade-off, it will be convenient to compare with other metrics based on the truncation parameter changes.

Original: When truncation parameter increases, one can presume that the diversity will be increased and likely the rarity of generations as in Figure 5.
Revised: When truncation parameter increases, one can presume that the diversity will be increased and likely the rarity of generations as in Figure 5.

Original: This is well shown in Figure 5a, where rarity score increases as the truncation parameter
Revised: This is well shown in Figure 5a, where rarity score increases as the truncation parameter

Original: R a r it y S
Revised: R a r it y S

Original: c
Revised: c

Original: o r e
Revised: o r e

Original: [Figure]
Revised: [Figure]

Original: s
Revised: s

Original: [Figure]
Revised: [Figure]

Original: S c
Revised: S c

Original: o r e
Revised: o r e

Original: [Figure]
Revised: [Figure]

Original: T r un ca ti on
Revised: T r un ca ti on

Original: T r
Revised: T r

Original: i c k 𝜓 = 0.
Revised: i c k 𝜓 = 0.

Original: 𝜓 = 1. 𝜓 = 1.
Revised: 𝜓 = 1. 𝜓 = 1.

Original: [Figure]
Revised: [Figure]

Original: increases.
Revised: increases.

Original: However, it is hard to be captured with FID or realism score, as (1) FID penalizes when the distribution is far from that of real as in the cases of ψ > 1 . 0 and (2) realism score is more focused on the fidelity.
Revised: However, it is hard to be captured with FID or realism score, as (1) FID penalizes when the distribution is far from that of real as in the cases of ψ > 1 . 0 and (2) realism score is more focused on the fidelity.

Original: On the other hand, as rarity score does not measure the fidelity directly, it will be helpful to use rarity score along with the fidelity metric, for selecting images or models.
Revised: On the other hand, as rarity score does not measure the fidelity directly, it will be helpful to use rarity score along with the fidelity metric, for selecting images or models.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we present the experimental results of the proposed metric for various standard GANs and datasets.
Revised: we present the experimental results of the proposed metric for various standard GANs and datasets.

Original: We use k = 3 for the manifold approximation throughout the experiments.
Revised: We use k = 3 for the manifold approximation throughout the experiments.

Original: We use 30k real images to approximate the real manifold and calculate the rarity of 10k fake images.
Revised: We use 30k real images to approximate the real manifold and calculate the rarity of 10k fake images.

Original: We use VGG16 as a feature extractor for all experiments except for Section 4.4.
Revised: We use VGG16 as a feature extractor for all experiments except for Section 4.4.

Original: 4. R ARITY OF I NDIVIDUAL G ENERATION
Revised: 4. R ARITY OF I NDIVIDUAL G ENERATION

Original: Figure 6 shows synthesized images with top 8 highest, top 8 lowest, and 8 middle rarity scores.
Revised: Figure 6 shows synthesized images with top 8 highest, top 8 lowest, and 8 middle rarity scores.

Original: As expected, the generated images with the lowest rarity scores are the ones that are typical in the training dataset.
Revised: As expected, the generated images with the lowest rarity scores are the ones that are typical in the training dataset.

Original: On the other hand, the generations with the highest rarity scores are far from typical images and have various characteristics and uniqueness that are rare in the training dataset.
Revised: On the other hand, the generations with the highest rarity scores are far from typical images and have various characteristics and uniqueness that are rare in the training dataset.

Original: For FFHQ dataset, for example, we can see clear and formal human face images in the ‘Bottom 8’ column whereas we can see elaborated face images such as face with a colorful makeup or a fancy hat in the ‘Top 8’ column.
Revised: For FFHQ dataset, for example, we can see clear and formal human face images in the ‘Bottom 8’ column whereas we can see elaborated face images such as face with a colorful makeup or a fancy hat in the ‘Top 8’ column.

Original: For AFHQ-Wild dataset, bottom images mostly show the faces of typical tigers while top images are mostly female lions with light-colored fur, which are far from the typical lion.
Revised: For AFHQ-Wild dataset, bottom images mostly show the faces of typical tigers while top images are mostly female lions with light-colored fur, which are far from the typical lion.

Original: Further, MetFaces dataset shows oil portraits in bottom images while top images show statues or flat paintings.
Revised: Further, MetFaces dataset shows oil portraits in bottom images while top images show statues or flat paintings.

Original: However, for the datasets which are uncurated and not aligned such as LSUN datasets, the top 8 images are not clear as that of the aligned datasets because the training dataset contains a lot of noisy images.
Revised: However, for the datasets which are uncurated and not aligned such as LSUN datasets, the top 8 images are not clear as that of the aligned datasets because the training dataset contains a lot of noisy images.

Original: S t y l e
Revised: S t y l e

Original: GAN 2 FF
Revised: GAN 2 FF

Original: HQ
Revised: HQ

Original: L S
Revised: L S

Original: UN - C a r
Revised: UN - C a r

Original: A F
Revised: A F

Original: HQ - W il d
Revised: HQ - W il d

Original: M e t F a ce s
Revised: M e t F a ce s

Original: Bottom
Revised: Bottom

Original: Top
Revised: Top

Original: C e l e b
Revised: C e l e b

Original: A - HQ
Revised: A - HQ

Original: Middle
Revised: Middle

Original: S t y l e
Revised: S t y l e

Original: GANADA
Revised: GANADA

Original: S t y l e
Revised: S t y l e

Original: GA
Revised: GA

Original: N
Revised: N

Original: NVA
Revised: NVA

Original: E
Revised: E

Original: A User Study on Rarity Score
Revised: A User Study on Rarity Score

Original: We have conducted a user study on whether our proposed rarity score aligns with the human perception of commonness.
Revised: We have conducted a user study on whether our proposed rarity score aligns with the human perception of commonness.

Original: 61 participants answered our survey.
Revised: 61 participants answered our survey.

Original: In this survey, we target on StyleGAN-FFHQ model.
Revised: In this survey, we target on StyleGAN-FFHQ model.

Original: The details and the results of user study are described in Figure 23a in Appendix I.
Revised: The details and the results of user study are described in Figure 23a in Appendix I.

Original: In total, 84% of answers were aligned with the rarity score, i.e., people interpret images with the lower rarity scores as more common images.
Revised: In total, 84% of answers were aligned with the rarity score, i.e., people interpret images with the lower rarity scores as more common images.

Original: For more challenging comparison between Group 10 (the highest group) and Group 5 (middle group), the alignment is lesser than that of Groupand Group 1 (the lowest group) and the portion of answer ”Both sets are similar” increases.
Revised: For more challenging comparison between Group 10 (the highest group) and Group 5 (middle group), the alignment is lesser than that of Groupand Group 1 (the lowest group) and the portion of answer ”Both sets are similar” increases.

Original: This suggests that the proposed metric can distinguish the degree of difference in commonness.
Revised: This suggests that the proposed metric can distinguish the degree of difference in commonness.

Original: 4. R ARITY OF G ENERATIVE M ODELS
Revised: 4. R ARITY OF G ENERATIVE M ODELS

Original: Not only the proposed metric can be applied on an individual generation, it can be used to compare generative models on the same dataset which shares the real manifold.
Revised: Not only the proposed metric can be applied on an individual generation, it can be used to compare generative models on the same dataset which shares the real manifold.

Original: While there exist diversity metrics on the generative models such as LPIPS (Zhang et al., 2018), our method is different in that it focuses on the ability to generate unique and rare generations while the existing metrics focus on diversity between generations.
Revised: While there exist diversity metrics on the generative models such as LPIPS (Zhang et al., 2018), our method is different in that it focuses on the ability to generate unique and rare generations while the existing metrics focus on diversity between generations.

Original: An Empirical Comparison between Generative Models We further compare the generative models using the rarity score.
Revised: An Empirical Comparison between Generative Models We further compare the generative models using the rarity score.

Original: Since our main focus is on the rare samples, we compare models by the RS-p score which is the mean rarity score of the top p % samples among 10000 samples.
Revised: Since our main focus is on the rare samples, we compare models by the RS-p score which is the mean rarity score of the top p % samples among 10000 samples.

Original: The left graph in Figure 7 shows the RS-p scores of the various models as p changes from 0.1 to 1.0.
Revised: The left graph in Figure 7 shows the RS-p scores of the various models as p changes from 0.1 to 1.0.

Original: The images in the right column of Figure 7 are the top 10 images with the highest scores, which corresponds to p = 0 .
Revised: The images in the right column of Figure 7 are the top 10 images with the highest scores, which corresponds to p = 0 .

Original: Under p = 0 .
Revised: Under p = 0 .

Original: 1 , StyleGAN2 shows the highest RS-p score even when compared with the real images as we can see in the generations on the right.
Revised: 1 , StyleGAN2 shows the highest RS-p score even when compared with the real images as we can see in the generations on the right.

Original: The images generated by StyleGAN2 are colorful and have diverse accessories, such as crown, mic, and hair band.
Revised: The images generated by StyleGAN2 are colorful and have diverse accessories, such as crown, mic, and hair band.

Original: On the other hand, the images from UnetGAN seem to have relatively less diversity.
Revised: On the other hand, the images from UnetGAN seem to have relatively less diversity.

Original: We can interpret that
Revised: We can interpret that

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: UnetGAN generates images more conservatively as its RS-p scores are the lowest among the models for all 0 .
Revised: UnetGAN generates images more conservatively as its RS-p scores are the lowest among the models for all 0 .

Original: 1 ≤ p ≤ 1 . 0 .
Revised: 1 ≤ p ≤ 1 . 0 .

Original: 4. C OMPARISONS BETWEEN S IMILAR D ATASETS
Revised: 4. C OMPARISONS BETWEEN S IMILAR D ATASETS

Original: We also propose a method to compare rarity between two different datasets which share the same concept, such as CelebA-HQ and FFHQ for human faces.
Revised: We also propose a method to compare rarity between two different datasets which share the same concept, such as CelebA-HQ and FFHQ for human faces.

Original: Rarity scores of each dataset are calculated on the union of two real datasets, Φ r =
Revised: Rarity scores of each dataset are calculated on the union of two real datasets, Φ r =

Original: Φ r 1 ∪ Φ r 2 .
Revised: Φ r 1 ∪ Φ r 2 .

Original: For example, when we calculate the scores of human face datasets such as CelebA-HQ and FFHQ, the real manifold manifold k ( Φ r ) is manifold k ( Φ CelebA − HQ ∪ Φ FFHQ ) .
Revised: For example, when we calculate the scores of human face datasets such as CelebA-HQ and FFHQ, the real manifold manifold k ( Φ r ) is manifold k ( Φ CelebA − HQ ∪ Φ FFHQ ) .

Original: Figure 8 shows comparisons on normalized rarity score for two sets of datasets that have the same concept.
Revised: Figure 8 shows comparisons on normalized rarity score for two sets of datasets that have the same concept.

Original: In the case of human face dataset, the density of
Revised: In the case of human face dataset, the density of

Original: CelebA-HQ is skewed left while the density of FFHQ is widely distributed to the right.
Revised: CelebA-HQ is skewed left while the density of FFHQ is widely distributed to the right.

Original: It means that FFHQ has more uncommon images than CelebA-HQ.
Revised: It means that FFHQ has more uncommon images than CelebA-HQ.

Original: In other words, there are less uncommon images in CelebA-HQ.
Revised: In other words, there are less uncommon images in CelebA-HQ.

Original: This makes sense because CelebA-HQ is collected among the celebrities while FFHQ is collected among the public with considerable variation in terms of age, ethnicity, and image background.
Revised: This makes sense because CelebA-HQ is collected among the celebrities while FFHQ is collected among the public with considerable variation in terms of age, ethnicity, and image background.

Original: The images in Figure 8 are top scoring images in each dataset.
Revised: The images in Figure 8 are top scoring images in each dataset.

Original: Since the maximum rarity score of FFHQ is greater than that of CelebA-HQ, the top images of FFHQ are relatively more uncommon.
Revised: Since the maximum rarity score of FFHQ is greater than that of CelebA-HQ, the top images of FFHQ are relatively more uncommon.

Original: Similarly, LSUN-Cat and AFHQ-Cat are both datasets on cat images where
Revised: Similarly, LSUN-Cat and AFHQ-Cat are both datasets on cat images where

Original: LSUN-Cat is uncurated cat images and AFHQ-Cat is curated with close-up cat faces only.
Revised: LSUN-Cat is uncurated cat images and AFHQ-Cat is curated with close-up cat faces only.

Original: In Figure 8, we can confirm that LSUN-Cat is more diverse than AFHQ-Cat as the rarity score distribution of
Revised: In Figure 8, we can confirm that LSUN-Cat is more diverse than AFHQ-Cat as the rarity score distribution of

Original: AFHQ-Cat is more on the left and concentrated compared to the distribution of AFHQ-Cat.
Revised: AFHQ-Cat is more on the left and concentrated compared to the distribution of AFHQ-Cat.

Original: 4. A S TUDY ON F EATURE E XTRACTORS
Revised: 4. A S TUDY ON F EATURE E XTRACTORS

Original: Since the proposed metric operates in the feature space, the perspective of the rarity may vary depending on which feature extractor is used.
Revised: Since the proposed metric operates in the feature space, the perspective of the rarity may vary depending on which feature extractor is used.

Original: Figure 9 presents images with high and low rarity scores
Revised: Figure 9 presents images with high and low rarity scores

Original: [Figure]
Revised: [Figure]

Original: for diverse feature extractors to verify the effects of the extractors.
Revised: for diverse feature extractors to verify the effects of the extractors.

Original: As a classifier trained on the
Revised: As a classifier trained on the

Original: Imagenet dataset, VGG16 focuses more on the objects and the patterns.
Revised: Imagenet dataset, VGG16 focuses more on the objects and the patterns.

Original: There are many images with colorful face painting among the high score samples with VGG16.
Revised: There are many images with colorful face painting among the high score samples with VGG16.

Original: On the other hand, ViT models such as DINO (Caron et al., 2021) are known to focus more on the structure compared to the CNN-based models (Park & Kim, 2022).
Revised: On the other hand, ViT models such as DINO (Caron et al., 2021) are known to focus more on the structure compared to the CNN-based models (Park & Kim, 2022).

Original: This can be seen from the second column of Figure 9.
Revised: This can be seen from the second column of Figure 9.

Original: The DINO-based high score images have various face angles while the low score images are mostly front-facing.
Revised: The DINO-based high score images have various face angles while the low score images are mostly front-facing.

Original: In the third column of Figure 9, the baby faces appear in the low score images, unlike the other two models which have formal adult images in the low score.
Revised: In the third column of Figure 9, the baby faces appear in the low score images, unlike the other two models which have formal adult images in the low score.

Original: This can be explained from the previous study that the CLIP model (Radford et al., 2021), which is a self-supervised image-text multi-modal model, can better capture semantics compared to the classifiers.
Revised: This can be explained from the previous study that the CLIP model (Radford et al., 2021), which is a self-supervised image-text multi-modal model, can better capture semantics compared to the classifiers.

Original: We conjecture that the model may consider the baby faces more similar between them than between the other images because baby faces are less distinctive and share a lot common properties compared to the adult faces.
Revised: We conjecture that the model may consider the baby faces more similar between them than between the other images because baby faces are less distinctive and share a lot common properties compared to the adult faces.

Original: Furthermore, we present a user study on the feature extractors in Figure 23b in Appendix I.
Revised: Furthermore, we present a user study on the feature extractors in Figure 23b in Appendix I.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a new evaluation metric called rarity score to measure how rare a synthesized image is based on the real data distribution.
Revised: In this paper, we propose a new evaluation metric called rarity score to measure how rare a synthesized image is based on the real data distribution.

Original: Each generated image is scored by estimating the density around the target image on the real manifold.
Revised: Each generated image is scored by estimating the density around the target image on the real manifold.

Original: Our rarity score enables several following tasks that existing metrics, such as FID and Precision&Recall, cannot do.
Revised: Our rarity score enables several following tasks that existing metrics, such as FID and Precision&Recall, cannot do.

Original: First, the proposed metric can evaluate the rarity or diversity of an individual image rather than a set of images.
Revised: First, the proposed metric can evaluate the rarity or diversity of an individual image rather than a set of images.

Original: Second, the proposed metric can be applied to compare between generative models or datasets that share the same concept in terms of the density of the rare samples.
Revised: Second, the proposed metric can be applied to compare between generative models or datasets that share the same concept in terms of the density of the rare samples.

Original: In addition, we provide studies on feature extractors that the viewpoint of commonness for each feature extractor varies depending on its structure (e.g. fully CNN, ViT) or how it was learned (e.g. fully supervised, self-supervised).
Revised: In addition, we provide studies on feature extractors that the viewpoint of commonness for each feature extractor varies depending on its structure (e.g. fully CNN, ViT) or how it was learned (e.g. fully supervised, self-supervised).

Original: The main limitation of our method is that it relies on the training dataset.
Revised: The main limitation of our method is that it relies on the training dataset.

Original: For example, for the LSUN datasets which have non-aligned images, noise images can have high rarity scores, which would not be what users expect to have strong characteristics for the target category.
Revised: For example, for the LSUN datasets which have non-aligned images, noise images can have high rarity scores, which would not be what users expect to have strong characteristics for the target category.

Original: Further, it might reflect the bias that the training dataset may contain.
Revised: Further, it might reflect the bias that the training dataset may contain.

Original: However, this can be useful for studying fairness and bias in the datasets or the generative models.
Revised: However, this can be useful for studying fairness and bias in the datasets or the generative models.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: It is challenging to quantify the creativity and is not exceptional for AI’s creativity.
Revised: It is challenging to quantify the creativity and is not exceptional for AI’s creativity.

Original: Rarity score that we propose can be regarded as a step toward quantifying creativity even if it focuses on uncommonness and uniqueness based on distribution distance.
Revised: Rarity score that we propose can be regarded as a step toward quantifying creativity even if it focuses on uncommonness and uniqueness based on distribution distance.

Original: To achieve better creativity quantification, it is essential to make a consensus among more multidisciplinary experts including artists, philosophers, social scientists as well as AI researchers.
Revised: To achieve better creativity quantification, it is essential to make a consensus among more multidisciplinary experts including artists, philosophers, social scientists as well as AI researchers.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We will release our code and the implementation details to be publicly available for reproducibility.
Revised: We will release our code and the implementation details to be publicly available for reproducibility.

Original: We will also include the url links to access various feature extractors.
Revised: We will also include the url links to access various feature extractors.

Original: A R ANK C ORRELATIONS
Revised: A R ANK C ORRELATIONS

Original: [Table]
Revised: [Table]

Original: φ g ∈ M(
Revised: φ g ∈ M(

Original: [Figure]
Revised: [Figure]

Original: StyleGAN2 LSUN
Revised: StyleGAN2 LSUN

Original: [Figure]
Revised: [Figure]

Original: Horse 1 2 3 4 5 6 7 8 9 10 k 0 .
Revised: Horse 1 2 3 4 5 6 7 8 9 10 k 0 .

Original: M e a n R a n k
Revised: M e a n R a n k

Original: C o rr e l a t i o n
Revised: C o rr e l a t i o n

Original: StyleGAN2-ADA MetFaces
Revised: StyleGAN2-ADA MetFaces

Original: φ g ∈ M 1 φ g ∈ M 2 φ g ∈ M 3 φ g ∈ M 4 φ g ∈ M 

Revised: φ g ∈ M 1 φ g ∈ M 2 φ g ∈ M 3 φ g ∈ M 4 φ g ∈ M

Original: φ g ∈ M 6 φ g ∈ Mφ g ∈ M 8 φ g ∈ M 9 φ g ∈ M(e) STG2-ADA MetFaces 1 2 3 4 5 6 7 8 9 10 k 0 .
Revised: φ g ∈ M 6 φ g ∈ Mφ g ∈ M 8 φ g ∈ M 9 φ g ∈ M(e) STG2-ADA MetFaces 1 2 3 4 5 6 7 8 9 10 k 0 .

Original: M e a n R a n k
Revised: M e a n R a n k

Original: C o rr e l a t i o n
Revised: C o rr e l a t i o n

Original: StyleGAN2-ADA AFHQ-Wild φ g ∈ M 1 φ g ∈ M 2 φ g ∈ M 3 φ g ∈ M 4 φ g ∈ M 5 φ g ∈ M 6 φ g ∈ Mφ g ∈ M 8 φ g ∈ M 9 φ g ∈ M(f) STG2-ADA AFHQ-Wild
Revised: StyleGAN2-ADA AFHQ-Wild φ g ∈ M 1 φ g ∈ M 2 φ g ∈ M 3 φ g ∈ M 4 φ g ∈ M 5 φ g ∈ M 6 φ g ∈ Mφ g ∈ M 8 φ g ∈ M 9 φ g ∈ M(f) STG2-ADA AFHQ-Wild

Original: B D ISTRIBUTIONS A CCORDING TO THE T RUNCATION P ARAMETER
Revised: B D ISTRIBUTIONS A CCORDING TO THE T RUNCATION P ARAMETER

Original: C C OMPARISONS BETWEEN M ODELS
Revised: C C OMPARISONS BETWEEN M ODELS

Original: R ea l
Revised: R ea l

Original: S t y l e GAN
Revised: S t y l e GAN

Original: S t y l e GAN 2 - ADA
Revised: S t y l e GAN 2 - ADA

Original: S t y l e GAN
Revised: S t y l e GAN

Original: U n e t GAN
Revised: U n e t GAN

Original: VQGAN
Revised: VQGAN

Original: D
Revised: D

Original: R EAL S AMPLES O RDERED BY NND AND F AKE
Revised: R EAL S AMPLES O RDERED BY NND AND F AKE

Original: S
Revised: S

Original: AMPLES O RDERED BY
Revised: AMPLES O RDERED BY

Original: R ARITY S CORE
Revised: R ARITY S CORE

Original: Lowest
Revised: Lowest

Original: R ea l
Revised: R ea l

Original: S a m p l e s (
Revised: S a m p l e s (

Original: NND )
Revised: NND )

Original: Highest
Revised: Highest

Original: F a k e
Revised: F a k e

Original: S a m p l e s ( R a r it y
Revised: S a m p l e s ( R a r it y

Original: S c o r e )
Revised: S c o r e )

Original: Lowest
Revised: Lowest

Original: R ea l S a m p l e s (
Revised: R ea l S a m p l e s (

Original: NND )
Revised: NND )

Original: Highest
Revised: Highest

Original: F a k e S a m p l e s ( R a r it
Revised: F a k e S a m p l e s ( R a r it

Original: y S c
Revised: y S c

Original: o r e )
Revised: o r e )

Original: Lowest
Revised: Lowest

Original: R ea l S a m p l e s (
Revised: R ea l S a m p l e s (

Original: NND )
Revised: NND )

Original: Highest
Revised: Highest

Original: F a k e S a m p l e s ( R a r it 
Revised: F a k e S a m p l e s ( R a r it y S c o r e )

Revised: Lowest

Revised: R ea l S a m p l e s (

Revised: NND )

Revised: Highest

Revised: F a k e S a m p l e s ( R a r it

Original: y S c
Revised: y S c

Original: o r e )
Revised: o r e )

Original: Lowest
Revised: Lowest

Original: R ea l S a m p l e s (
Revised: R ea l S a m p l e s (

Original: NND )
Revised: NND )

Original: Highest
Revised: Highest

Original: F a k e S a m p l e s ( R a r it
Revised: F a k e S a m p l e s ( R a r it

Original: y S c
Revised: y S c

Original: o r e )
Revised: o r e )

Original: Lowest
Revised: Lowest

Original: R ea l S a m p l e s (
Revised: R ea l S a m p l e s (

Original: NND )
Revised: NND )

Original: Highest
Revised: Highest

Original: F a k e S a m p l e s ( R a r ity S c o r e )
Revised: F a k e S a m p l e s ( R a r it

Original: Lowest

Original: R ea l S a m p l e s (

Original: NND )

Original: Highest

Original: F a k e S a m p l e s ( R a r it

Original: y S c
Revised: y S c

Original: o r e )
Revised: o r e )

Original: E E XAMPLES OF O UT - OF -M ANIFOLD S AMPLES
Revised: E E XAMPLES OF O UT - OF -M ANIFOLD S AMPLES

Original: FF
Revised: FF

Original: HQ
Revised: HQ

Original: L S
Revised: L S

Original: UNC a r
Revised: UNC a r

Original: L S
Revised: L S

Original: UNC a t
Revised: UNC a t

Original: L S
Revised: L S

Original: UNH
Revised: UNH

Original: o r s e
Revised: o r s e

Original: M e t f ace s
Revised: M e t f ace s

Original: A F
Revised: A F

Original: HQW il d
Revised: HQW il d

Original: FF
Revised: FF

Original: HQ
Revised: HQ

Original: M e t F ace s
Revised: M e t F ace s

Original: S t y l e
Revised: S t y l e

Original: GAN
Revised: GAN

Original: S t y l e
Revised: S t y l e

Original: GAN 2ADA
Revised: GAN 2ADA

Original: S t y l e
Revised: S t y l e

Original: GAN
Revised: GAN

Original: F
Revised: F

Original: R ESULTS FOR THE V ARIOUS F EATURE E XTRACTORS
Revised: R ESULTS FOR THE V ARIOUS F EATURE E XTRACTORS

Original: [Figure]
Revised: [Figure]

Original: G A DDITIONAL E XPERIMENTS ON THE S ELECTION OF k
Revised: G A DDITIONAL E XPERIMENTS ON THE S ELECTION OF k

Original: [Figure]
Revised: [Figure]

Original: a l
Revised: a l

Original: [Figure]
Revised: [Figure]

Original: e N e g a t
Revised: e N e g a t

Original: i v e
Revised: i v e

Original: [Figure]
Revised: [Figure]

Original: a t
Revised: a t

Original: [Figure]
Revised: [Figure]

Original: F l
Revised: F l

Original: N g a t i v
Revised: N g a t i v

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H R ESULTS FOR NON -GAN- BASED M ODELS
Revised: H R ESULTS FOR NON -GAN- BASED M ODELS

Original: [Figure]
Revised: [Figure]

Original: I
Revised: I

Original: U SER S TUDY
Revised: U SER S TUDY

Revised: Total

Revised: Group 10 vs

Revised: Group 10 vs

Revised: Aligned

Revised: Not aligned

Original: [Figure]
Revised: Similar VGG16 62.0% DINO 9.2% CLIP 28.9% (

Revised: Preference between various backbone models

Revised: Rarity vs Realism

Revised: StyleGAN2 vs UnetGAN

Original:  We have conducted user study whether our proposed rarity score aligns with the human perception of commonness and which backbone model is preferred.
Revised: Aligned Not aligned We have conducted user study whether our proposed rarity score aligns with the human perception of commonness and which backbone model is preferred.

Original: 61 participants answered to our survey.
Revised: 61 participants answered to our survey.

Original: In this survey, we target on StyleGAN-FFHQ model.
Revised: In this survey, we target on StyleGAN-FFHQ model.

Original: I. H UMAN E VALUATION FOR THE C OMMMONNESS.
Revised: I. H UMAN E VALUATION FOR THE C OMMMONNESS

Original: To investigate whether the proposed rarity score aligns with the human perception of commonness/uncommonness, we asked the participants to choose more typical and common images between two sets of images with different levels of rarity scores.
Revised: To investigate whether the proposed rarity score aligns with the human perception of commonness/uncommonness, we asked the participants to choose more typical and common images between two sets of images with different levels of rarity scores.

Original: Each set contains 9 images.
Revised: Each set contains 9 images.

Original: The participants can answer either ”A”, ”B”, or ”Similar”.
Revised: The participants can answer either ”A”, ”B”, or ”Similar”.

Original: We divided the generated samples into 10 groups according to its rarity score.
Revised: We divided the generated samples into 10 groups according to its rarity score.

Original: Group 1 has the lowest rarity scores and Group 10 has the highest rarity scores.
Revised: Group 1 has the lowest rarity scores and Group 10 has the highest rarity scores.

Original: Each participants are asked to answer 10 sets of choices, 5 sets comparing Group 10 and
Revised: Each participants are asked to answer 10 sets of choices, 5 sets comparing Group 10 and

Original: Group 1 and 5 sets comparing Group 10 and Group 5.
Revised: Group 1 and 5 sets comparing Group 10 and Group 5.

Original: The order of sets are shuffled and the location of Group 10 (left or right) is also randomly shuffled.
Revised: The order of sets are shuffled and the location of Group 10 (left or right) is also randomly shuffled.

Original: The results are summarized in Figure 23a.
Revised: The results are summarized in Figure 23a.

Original: In total, 84% of answers were aligned with the rarity score, i.e., human perception of commonness matches with the low rarity scores.
Revised: In total, 84% of answers were aligned with the rarity score, i.e., human perception of commonness matches with the low rarity scores.

Original: For more challenging comparison between Group 10 and Group 5, the alignment is lesser than that of Groupand Group 1 and the portion of answer ”Similar” increases.
Revised: For more challenging comparison between Group 10 and Group 5, the alignment is lesser than that of Groupand Group 1 and the portion of answer ”Similar” increases.

Original: This suggests that the proposed metric can distinguish the degree of difference in commonness.
Revised: This suggests that the proposed metric can distinguish the degree of difference in commonness.

Original: I. H UMAN E VALUATION FOR THE C OMMMONNESS .
Revised: I. P REFERENCE ON B ACKBONE

Original: We further asked to the participants which backbone is preferred.
Revised: We further asked to the participants which backbone is preferred.

Original: For this survey, each participant is given 3 sets of images with different backbones; VGG16, DINO, and CLIP.
Revised: For this survey, each participant is given 3 sets of images with different backbones; VGG16, DINO, and CLIP.

Original: Each set containsimages and the order of backbone models is shuffled each time.
Revised: Each set containsimages and the order of backbone models is shuffled each time.

Original: Each participant answers 5 times for this type of questions.
Revised: Each participant answers 5 times for this type of questions.

Original: As a result, VGG16 is the most preferred with 62% of times.
Revised: As a result, VGG16 is the most preferred with 62% of times.

Original: CLIP was the second most preferred backbone, but the gap between VGG16 and CLIP was significant.
Revised: CLIP was the second most preferred backbone, but the gap between VGG16 and CLIP was significant.

Original: J N OISE R OBUSTNESS OF R ARITY S CORE
Revised: I. C OMPARISON WITH R EALISM S CORE

Revised: We additionally conduct a user study on which metric is preferred for measuring uncommonness between rarity score and realism score.

Revised: We give 10 sets of images, where each set contains top 9 and bottom 9 images for each of the rarity score and realism score.

Revised: The location (left or right) between rarity score and realism score is randomly selected for each set.

Revised: For a fairer comparison, we excluded out-of-manifold samples from the samples of the top/bottom realism scores.

Revised: We ask each participant that which set of images represents rare/common images best.

Revised: 30 participants answered to this survey.

Revised: The result is summarized in Figure 23c.

Revised: In total, 84% of answers choose the image set from rarity score compared to the image set of realism score.

Revised: This supports that rarity score is better suitable for distinguishing and measuring uncommonness realism score, while realism score is useful for measuring fidelity of individual generation.

Revised: I. A LIGNMENT WITH RS- P

Revised: We further ask participants which model is preferred in terms of rarity between GAN models trained on FFHQ.

Revised: We choose two GAN models, StyleGAN2 and UnetGAN for the comparison, which have the highest RS-p score and the lowest RS-p score, respectively.

Revised: We show 10 sets of images, where each set contains top 9 images with the rarity score for each of the StyleGAN2 and UnetGAN.

Revised: The result is summarized in Figure 23c.

Revised: ‘Aligned’ represents that StyleGAN2 is chosen to have rarer images, which is aligned with RS-p score as StyleGAN2 has higher RS-p score compared to

Revised: UnetGAN.

Revised: As a result, 68% of answers were aligned with RS-p score.

Revised: While it is difficult to clearly specify the order of rarity from the generations in Figure 7, the result of user study suggests that

Revised: RS-p score can guide to a model with more rare generations.

Revised: J N OISE R OBUSTNESS OF R ARITY S CORE

Original: [Figure]
Revised: [Figure]

Original: R a n k
Revised: R a n k

Original: C o rr e l a t i o n
Revised: C o rr e l a t i o n

Original: [Figure]
Revised: [Figure]

Original: K RS- P OF FFHQ M ODELS WITH V ARIOUS B ACKBONES
Revised: K RS- P OF FFHQ M ODELS WITH V ARIOUS B ACKBONES

Original: [Figure]
Revised: [Figure]

Original: We compute RS-p scores of various FFHQ models by adapting the backbone to VGG16, DINO and
Revised: We compute RS-p scores of various FFHQ models by adapting the backbone to VGG16, DINO and

Original: CLIP.
Revised: CLIP.

Original: Figure 25 shows the results.
Revised: Figure 25 shows the results.

Original: As different backbones have different viewpoints of rarity, the order of RS-p also depends on the backbone.
Revised: As different backbones have different viewpoints of rarity, the order of RS-p also depends on the backbone.

Original: For VGG16, StyleGAN2 shows the best RS-p score for its colorful decorations in the generations.
Revised: For VGG16, StyleGAN2 shows the best RS-p score for its colorful decorations in the generations.

Original: On the other hand, for DINO, VQGAN shows the highest RS-p score which have some artifacts on the corner of the image.
Revised: On the other hand, for DINO, VQGAN shows the highest RS-p score which have some artifacts on the corner of the image.

Original: This is because for DINO, the image with objects other than a human face is regarded as rare.
Revised: This is because for DINO, the image with objects other than a human face is regarded as rare.

Original: For CLIP, VQGAN again shows the best RS-p score.
Revised: For CLIP, VQGAN again shows the best RS-p score.

Original: However, for this time, boys with curly hair have the highest scores.
Revised: However, for this time, boys with curly hair have the highest scores.

Original: It turns out that, a real image of a magazine cover of a teenage star has the highest nearest neighbor distance and the generations similar to this image get high rarity scores for CLIP backbone. 
Revised: It turns out that, a real image of a magazine cover of a teenage star has the highest nearest neighbor distance and the generations similar to this image get high rarity scores for CLIP backbone. L A PPLICATION ON A T EXT - TO -I MAGE M ODEL [Figure]

Revised: (sorted by the occurrence in top 10) 79 a brown dog standing next to a toilet in a bathroom.

Revised: 67 an adult is cooking at the kitchen counter near a baby and a dog.

Revised: 48 a man in the kitchen standing with his dog.

Revised: 12 a dog is standing next to a toilet playing with tissue 57 three people stand on the street talking next to a dog.

Revised: No

Revised: [Figure]

Revised: Prompt (sorted by the occurrence in bottom 10) 62 the small dog has a tiny black nose.

Revised: 24 an image of a dog sitting on a sitting 64 a dog looking away from the camera in the wind.

Revised: 63 a grey dog with a black collar sits outside in the sun.

Revised: 83 a close up of a dog near a bowl

Revised: [Figure]

Revised: We compute the rarity scores on a LDM (Latent Diffusion Model), a recently released text-to-image model.

Revised: For the experiments, we use stable-diffusion-v1-3 from the official git repository.

Revised: To show an example usage of the proposed metric to a text-to-image models, we sample 2000 real images which contain ”dog” in its caption, from LAION-400M dataset, and use 100 prompts containing ”dog” from MS COCO 2017 validation dataset for generating fake samples.

Revised: For each prompt, 10 images are generated from the model.

Revised: We compute the rarity scores and bottom 10 and top 10 images are shown in Figure 26.

Revised: We can see that the closed-up dog faces have low rarity scores, which can be regarded as typical dog images.

Revised: On the other hand, dog images with various backgrounds or environments have high rarity scores.

Revised: Prompts on the right side of Figure 26 can give some guides to the users of text-to-image models, for example, which prompts to use for generating a more unique and fun images.

Revised: M D ENSITIES OF C OMMON AND R ARE S AMPLES

Revised: [Table]

Revised: We apply the kernel density estimation (KDE) on the real features, to numerically show that the common samples with low NND in Figure 1 have high density in the feature space and the rare samples with high NND have low density in the feature space.

Revised: We use Gaussian kernel with the bandwidth 0.2.

Revised: For low NND and high NND, we use 1% of the total real samples, respectively.

Revised: Table 1 summarizes the mean of log densities for each set of samples.

Revised: In all datasets, we can see that the samples with low NND, which look typical and common, have significantly higher density compared to the samples with high NND, which have strong as shown in Figure 1.