Original: R OBUST M ANIFOLD E STIMATION A PPROACH FOR E VALUATING F IDELITY AND D IVERSITY
Revised: R OBUST M ANIFOLD E STIMATION A PPROACH FOR E VALUATING F IDELITY AND D IVERSITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a robust and reliable evaluation metric for generative models by introducing topological and statistical treatments for a rigorous support manifold estimation.
Revised: We propose a robust and reliable evaluation metric for generative models by introducing topological and statistical treatments for a rigorous support manifold estimation.

Original: Existing metrics, such as Inception Score (IS), Fréchet Inception
Revised: Existing metrics, such as Inception Score (IS), Fréchet Inception

Original: Distance (FID), and the variants of Precision and Recall ( P&R ), heavily rely on support manifolds that are estimated from sample features.
Revised: Distance (FID), and the variants of Precision and Recall ( P&R ), heavily rely on support manifolds that are estimated from sample features.

Original: However, the reliability of their estimation has not been seriously discussed (and overlooked) even though the quality of the evaluation entirely depends on it.
Revised: However, the reliability of their estimation has not been seriously discussed (and overlooked) even though the quality of the evaluation entirely depends on it.

Original: In this paper, we propose Topological Precision and Recall ( TopP&R , pronounced “topper”), which provides a systematic approach to estimating support manifolds, retaining only topologically and statistically important features with a certain level of confidence.
Revised: In this paper, we propose Topological Precision and Recall ( TopP&R , pronounced “topper”), which provides a systematic approach to estimating support manifolds, retaining only topologically and statistically important features with a certain level of confidence.

Original: This not only makes TopP&R strong for noisy features, but also provides statistical consistency.
Revised: This not only makes TopP&R strong for noisy features, but also provides statistical consistency.

Original: Our theoretical and experimental results show that TopP&R is robust to outliers and non-independent and identically distributed (Non-IID) perturbations, while accurately capturing the true trend of change in samples.
Revised: Our theoretical and experimental results show that TopP&R is robust to outliers and non-independent and identically distributed (Non-IID) perturbations, while accurately capturing the true trend of change in samples.

Original: To the best of our knowledge, this is the first evaluation metric focused on the robust estimation of the support manifold and provides its statistical consistency under noise.
Revised: To the best of our knowledge, this is the first evaluation metric focused on the robust estimation of the support manifold and provides its statistical consistency under noise.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In keeping with the remarkable improvements of deep generative models (Karras et al., 2019; 2020; 2021;
Revised: In keeping with the remarkable improvements of deep generative models (Karras et al., 2019; 2020; 2021;

Original: Brock et al., 2018; Ho et al., 2020; Kingma & Welling, 2013; Sauer et al., 2022; 2021; Kang &
Revised: Brock et al., 2018; Ho et al., 2020; Kingma & Welling, 2013; Sauer et al., 2022; 2021; Kang &

Original: Park, 2020), evaluation metrics that can well measure the performance of generative models have also been continuously developed (Salimans et al., 2016;
Revised: Park, 2020), evaluation metrics that can well measure the performance of generative models have also been continuously developed (Salimans et al., 2016;

Original: Heusel et al., 2017; Sajjadi et al., 2018;
Revised: Heusel et al., 2017; Sajjadi et al., 2018;

Original: Kynkäänniemi et al., 2019; Naeem et al., 2020).
Revised: Kynkäänniemi et al., 2019; Naeem et al., 2020).

Original: For instance, Inception Score (IS) (Salimans et al., 2016) measures the Kullback-Leibler divergence between the real and fake sample distributions in
Revised: For instance, Inception Score (IS) (Salimans et al., 2016) measures the Kullback-Leibler divergence between the real and fake sample distributions.

Original: VGG (Simonyan & Zisserman, 2014) feature space.

Original: Fréchet Inception Score (FID)(Heusel et al., 2017) calculates the distance between the real and fake support manifolds using the estimated mean and variance under the multi-Gaussian assumption.
Revised: Fréchet Inception Score (FID)(Heusel et al., 2017) calculates the distance between the real and fake support manifolds using the estimated mean and variance under the multi-Gaussian assumption.

Original: The original Precision and Recall( P&R ) (Sajjadi et al., 2018) and its variants (Kynkäänniemi et al., 2019; Naeem et al., 2020) were the first to separately investigate the fidelity and diversity aspects of generative performance.
Revised: The original Precision and Recall (Sajjadi et al., 2018) and its variants (Kynkäänniemi et al., 2019; Naeem et al., 2020) measure the fidelity and diversity by investigating whether the generated image belongs to the real image distribution and the generative model can reproduce all the real images in the distribution, respectively.

Original: Considering the eminent progress of deep generative models based on these existing metrics, some may question why we need another evaluation study.
Revised: Considering the eminent progress of deep generative models based on these existing metrics, some may question why we need another evaluation study.

Original: In this paper, we argue that we need more reliable evaluation metrics now precisely, because deep generative models have reached sufficient maturity.
Revised: In this paper, we argue that we need more reliable evaluation metrics now precisely, because deep generative models have reached sufficient maturity.

Original: To provide a more accurate and comprehensive ideas and to illuminate a new direction of improvements in the generative field, we need a more robust and reliable evaluation metric.
Revised: To provide a more accurate and comprehensive ideas and to illuminate a new direction of improvements in the generative field, we need a more robust and reliable evaluation metric.

Original: In fact, it has been recently reported that even the most widely used evaluation metric, FID, sometimes doesn’t match with the expected perceptual quality, fidelity, and diversity, which means the metrics are not always working properly (Kynkäänniemi et al., 2022).
Revised: In fact, it has been recently reported that even the most widely used evaluation metric, FID, sometimes doesn’t match with the expected perceptual quality, fidelity, and diversity, which means the metrics are not always working properly (Kynkäänniemi et al., 2022).

Original: In addition to this, in practice, not only do generated samples but also real data in the wild often contain lots of artifacts, and these have been shown to seriously perturb the existing evaluation metrics, giving a false sense of improvements (Naeem et al., 2020; Kynkäänniemi et al., 2022).
Revised: In addition to this, in practice, not only do generated samples but also real data in the wild often contain lots of artifacts (Pleiss et al., 2020; Li et al., 2022), and these have been shown to seriously perturb the existing evaluation metrics, giving a false sense of improvements (Naeem et al., 2020; Kynkäänniemi et al., 2022).

Original: An ideal evaluation metric must capture the real signal of the data, while being robust to noise.
Revised: An ideal evaluation metric must capture the real signal of the data, while being robust to noise.

Original: Note that there is an inherent tension in developing metrics that meets these goals.
Revised: Note that there is an inherent tension in developing metrics that meets these goals.

Original: On one hand, the metric should be sensitive enough so that it can capture real signals lurking in data.
Revised: On one hand, the metric should be sensitive enough so that it can capture real signals lurking in data.

Original: On the other hand, itmust ignore noises that hide the signal. However, sensitive metrics are inevitably susceptible to noise [Figure] (𝑃
Revised: On the other hand, it

Original: [Figure]
Revised: [Figure]

Original: 𝑠𝑢𝑝𝑝(𝑄)

Original: [Figure] [Figure] [Figure] ̂𝑝 !
Revised: must ignore noises that hide the signal.

Original: (#) ̂𝑝 !

Original: (%) ̂𝑝 ! (&) 𝜃 [Figure] ̂𝑝 !
Revised: However, sensitive metrics are inevitably susceptible to noise to some extent.

Original: − ̂𝑝 ! [Figure] [Figure] ̂𝑝 !
Revised: To address this

Original: − ̂𝑝 !

Original: [Figure]
Revised: [Figure]

Original: to some extent. To address this, one needs a systematic way to answer the following two questions: 1) what is signal and what is noise? and 2) how do we draw a line between them?
Revised: one needs a systematic way to answer the following two questions: 1) what is signal and what is noise? and 2) how do we draw a line between them?

Original: One solution can be to use the idea of statistical inference and topological data analysis (TDA).
Revised: One solution can be to use the idea of statistical inference and topological data analysis (TDA).

Original: Topological data analysis (TDA) (Carlsson, 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data.
Revised: Topological data analysis (TDA) (Carlsson, 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data.

Original: A key object in TDA is persistent homology, whichquantifies salient topological features of data by observing them in multi-resolutions. It observes how long each homological feature would survive across different resolutionsthrough the lens of topology and provides a way to quantify its importance; That is, if some features persist longer than others, we considerthe features and the data that compose them asa topological signal and vice versa as noise.Then, statistical inference provides a systematic way to establish statistical thresholds that separate the topological signal from topological noise with statistical interpretation.
Revised: A key object in TDA is persistent homology, which observes how long each topological feature would survive over varying resolutions and provides a measure to quantify its significance; i.e. , if some features persist longer than others over varying resolutions, we consider them as topological signal and vice versa as noise.

Original: In this paper, we propose to combine these ideas to form a more robust and compact feature manifold and overcome various issues from the conventional metrics.
Revised: In this paper, we propose to combine these ideas to form a more robust and compact feature manifold and overcome various issues from the conventional metrics.

Original: Our main contributions are as follows: we introduce (1) an approach to directly estimate a support manifold via Kernel Density Estimator (KDE) derived under topological conditions; (2) a new metric that is robust to outliers while reliably detecting the change of distributions on various scenarios; and (3) a theoretical guarantee of consistency with robustness under very weak assumptions that is suitable for high dimensional data.
Revised: Our main contributions are as follows: we introduce (1) an approach to directly estimate a support manifold via Kernel Density Estimator (KDE) derived under topological conditions; (2) a new metric that is robust to outliers while reliably detecting the change of distributions on various scenarios; and (3) a theoretical guarantee of consistency with robustness under very weak assumptions that is suitable for high dimensional data; (4) combining a noise framework and a statistical inference in TDA—consistencies under noise framework have studied in much literature, but not quite in geometrical or topological setting.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: To lay the foundation for our theoretical analysis, we introduce the main idea of persistent homology and its confidence estimation techniques that bring the benefit of using topological and statistical tools for addressing uncertainty in samples.
Revised: To lay the foundation for our theoretical analysis, we introduce the main idea of persistent homology and its confidence estimation techniques that bring the benefit of using topological and statistical tools for addressing uncertainty in samples.

Original: In later sections, we use these tools to analyze the effects of outliers in evaluating generative models and provide more rigorous way of scoring the samples based on the confidence level we set.
Revised: In later sections, we use these tools to analyze the effects of outliers in evaluating generative models and provide more rigorous way of scoring the samples based on the confidence level we set.

Original: For space reasons, we only provide a brief overview of the concepts that are relevant to this work and refer the reader to Section A or (Edelsbrunner & Harer, 2010; Chazal & Michel, 2021; Wasserman, 2018; Hatcher, 2002) for further details.
Revised: For space reasons, we only provide a brief overview of the concepts that are relevant to this work and refer the reader to Appendix A or (Edelsbrunner & Harer, 2010; Chazal & Michel, 2021; Wasserman, 2018; Hatcher, 2002) for further details.

Original: 2. N OTATION
Revised: 2. N OTATION

Original: For any x and r > 0 , we use the notation B d ( x, r )
Revised: For any x and r > 0 , we use the notation B d ( x, r )

Original: = { y : d ( y, x ) < r } be the open ball in distance d of radius r .
Revised: = { y : d ( y, x ) < r } be the open ball in distance d of radius r .

Original: We also write B ( x, r ) when d is understood from context.
Revised: We also write B ( x, r ) when d is understood from context.

Original: For a distribution P on R d , we let supp( P ) :
Revised: For a distribution P on R d , we let supp( P ) :

Original: = { x ∈ R d : P ( B ( x, r )) > 0 for all r > 0 } be the support of P .
Revised: = { x ∈ R d : P ( B ( x, r )) > 0 for all r > 0 } be the support of P .

Original: Throughout the paper, we refer to supp(P) as support manifold of P , or simply support, or manifold, but we don’t necessarily require the (geometrical) manifold structure on supp(P) .
Revised: Throughout the paper, we refer to supp(P) as support manifold of P , or simply support, or manifold, but we don’t necessarily require the (geometrical) manifold structure on supp(P) .

Original: For a kernel function
Revised: For a kernel function

Original: K : R d → R , a dataset X =
Revised: K : R d → R , a dataset X =

Original: { X 1 , . . . ,
Revised: { X 1 , . . . ,

Original: X n } ⊂ R d and bandwidth h > 0 , we let the kernel density estimator (KDE) as ˆ p h ( x ) :
Revised: X n } ⊂ R d and bandwidth h > 0 , we let the kernel density estimator (KDE) as ˆ p h ( x ) :

Original: = 1 nh d (cid:80) ni =1 K (cid:0) x − X i h (cid:1) , and we let the average KDE as p h :=
Revised: = 1 nh d (cid:80) ni =1 K (cid:0) x − X i h (cid:1) , and we let the average KDE as p h :=

Original: E [ˆ p h ] .
Revised: E [ˆ p h ] .

Original: We denote by P , Q the probability distributions in R d of real data and generated samples, respectively.
Revised: We denote by P , Q the probability distributions in R d of real data and generated samples, respectively.

Original: And we use X = { X 1 , . . .
Revised: And we use X = { X 1 , . . .

Original: X n } ⊂ R d and Y =
Revised: X n } ⊂ R d and Y =

Original: { Y 1 , . . .
Revised: { Y 1 , . . .

Original: , Y m } ⊂ R d for real data and generated samples possibly with noise, respectively.
Revised: , Y m } ⊂ R d for real data and generated samples possibly with noise, respectively.

Original: 2. C ONFIDENCE BAND ESTIMATION
Revised: 2. C ONFIDENCE BAND ESTIMATION

Original: Statistical inference has recently been developed for topological data analysis (Chazal et al., 2013; 2015; Fasy et al., 2014).
Revised: Statistical inference has recently been developed for topological data analysis (Chazal et al., 2013; 2015; Fasy et al., 2014).

Original: Topological data analysis consists of features reflecting topological characteristics of data, and it is of question to distinguish features that are indeed from geometrical structures and features that are insignificant or due to noise.
Revised: Topological data analysis consists of features reflecting topological characteristics of data, and it is of question to distinguish features that are indeed from geometrical structures and features that are insignificant or due to noise.

Original: To statistically separate topologically significant features from topological noise, we use a confidence band.
Revised: To statistically separate topologically significant features from topological noise, we use a confidence band.

Original: Given the significance level α , let confidence band
Revised: Given the significance level α , let confidence band

Original: c
Revised: c

Original: X be the bootstrap bandwidth of ∥ ˆ p h −  p ∗ h ∥ ∞ .
Revised: X be the bootstrap bandwidth of ∥ ˆ p h − ˆ p ∗ h ∥ ∞ .

Original: Then it satisfies lim inf n →∞ P ( ∥ ˆ p
Revised: Then it satisfies lim inf n →∞ P ( ∥ ˆ p

Original: h − p h
Revised: h − p h

Original: ∥ ∞ < c X ) ≥ 1 − α , as in Proposition 4 in Section C.
Revised: ∥ ∞ < c X ) ≥ 1 − α , as in Proposition 4 in Appendix C.

Original: This confidence band can be used to determine simultaneously significant topological features while filtering out noise features.
Revised: This confidence band can be used to determine simultaneously significant topological features while filtering out noise features.

Original: The algorithm for computing c X is described below. 
Revised: The algorithm for computing c X is described below. density 11:

Original:  θ with √ n || ˆ p h − ˆ p ∗ h || ∞ 13: end for 14: # grid search for the confidence band 15: for q ∈ [ min (ˆ θ ) , max (ˆ θ )] do 16: count =17: for element ∈
Revised: = Append ˆ θ with √ n || ˆ p h − ˆ p ∗ h || ∞ 13: end for 14: # grid search for the confidence band 15: for q ∈ [ min (ˆ θ ) , max (ˆ θ )] do 16: count =17: for element ∈

Original: ˆ θ do 18: # count significant difference 19: if element > q then 20: count = count +21: end if 22: end for 23:
Revised: ˆ θ do 18: # count significant difference 19: if element > q then 20: count = count +21: end if 22: end for 23:

Original: # define the band threshold 24: if count /k
Revised: # define the band threshold 24: if count /k

Original: ≈
Revised: ≈

Original: α
Revised: α

Original: then 25: q α = q 26: end if 27: end for 28: # define estimated confidence band 29: c α = q α / √ n
Revised: then 25: q α = q 26: end if 27: end for 28: # define estimated confidence band 29: c α = q α / √ n

Original:  R OBUST SUPPORT MANIFOLD ESTIMATION FOR RELIABLE EVALUATION
Revised:  R OBUST SUPPORT MANIFOLD ESTIMATION FOR RELIABLE EVALUATION

Original: Current evaluation metrics for generative models typically rely on strong regularity conditions.
Revised: Current evaluation metrics for generative models typically rely on strong regularity conditions.

Original: For example, they assume samples are well-curated without outliers or adversarial perturbation, real or generative models have bounded densities, etc.
Revised: For example, they assume samples are well-curated without outliers or adversarial perturbation, real or generative models have bounded densities, etc.

Original: However, practical scenarios are wild: both real and generated samples can be corrupted with noise from various sources, and the real data can be very sparsely distributed without density.
Revised: However, practical scenarios are wild: both real and generated samples can be corrupted with noise from various sources, and the real data can be very sparsely distributed without density.

Original: In this work, we consider more general and practical situations, wherein both real and generated samples can have noises that come from sampling procedure, remained uncertainty due to data or model, etc. 
Revised: In this work, we consider more general and practical situations, wherein both real and generated samples can have noises that come from sampling procedure, remained uncertainty due to data or model, etc. For more detailed discussions on the philosophy of our metric, please see Appendix E.

Original: 3. T OPOLOGICAL PRECISION AND RECALL
Revised: 3. T OPOLOGICAL PRECISION AND RECALL

Original: In the ideal case where we have full access to the probability distributions P and Q , we define the precision and the recall of distributions as precisionP ( Q ) : = Q (supp( P )) , recall Q ( P ) :
Revised: In the ideal case where we have full access to the probability distributions P and Q , we define the precision and the recall of distributions as precision

Original: = P (supp( Q )) .
Revised: [Equation]

Original: These correspond to the max precision and the max recall in Sajjadi et al.
Revised: These correspond to the max precision and the max recall in Sajjadi et al.

Original: We tweak the precision as precision P ( Y ) =
Revised: We tweak the precision as precision P ( Y ) =

Original: Q (supp( P ) ∩ supp( Q )) /Q
Revised: Q (supp( P ) ∩ supp( Q )) /Q

Original: (supp( Q )) , and define the precision of data points as precision P ( Y [Equation] :=
Revised: (supp( Q )) , and define the precision of data points as precision P ( Y ) :=

Original: [Equation]
Revised: [Equation]

Original: which is just replacing the distribution Q by the empirical distribution 1 m (cid:80) mj
Revised: which is just replacing the distribution Q by the empirical distribution 1 m (cid:80) mj

Original: =1 δ Y j of Y in the precision.
Revised: =1 δ Y j of Y in the precision.

Original: We similarly define the recall of data points as
Revised: We similarly define the recall of data points as

Original: [Equation]
Revised: [Equation]

Original: However, in practice, supp( P ) and supp( Q ) are not known a priori and need to be estimated, and since we allow noise, these estimates should be robust to noise.
Revised: However, in practice, supp( P ) and supp( Q ) are not known a priori and need to be estimated, and since we allow noise, these estimates should be robust to noise.

Original: For this, we use the kernel density estimator (KDE) and the bootstrap bandwidth to robustly estimate the support.
Revised: For this, we use the kernel density estimator (KDE) and the bootstrap bandwidth to robustly estimate the support.

Original: Given h n > 0 and a significance level α ∈ (0 , 1) , we use the KDE ˆ p h n ( x ) :
Revised: Given h n > 0 and a significance level α ∈ (0 , 1) , we use the KDE ˆ p h n ( x ) :

Original: = 1 nh dn (cid:80) ni =1 K (cid:16) x − X i h n (cid:17) of X , and we use the bootstrap bandwidth c X of (cid:13)(cid:13)
Revised: = 1 nh dn (cid:80) ni =1 K (cid:16) x − X i h n (cid:17) of X , and we use the bootstrap bandwidth c X of (cid:13)(cid:13)

Original: ˆ p h n − ˆ p ∗ h n (cid:13)(cid:13) ∞ from Section 2.
Revised: ˆ p h n − ˆ p ∗ h n (cid:13)(cid:13) ∞ from Section 2.

Original: Then we estimate the support of P by the superlevel set at c X as ˆ supp
Revised: Then we estimate the support of P by the superlevel set at c X as ˆ supp

Original: [Equation]
Revised: [Equation]

Original: P ) =

Original: [Equation]

Original: h n [ c X , ∞ ) .
Revised: [ c X , ∞ ) .

Original: Similarly, we let ˆ q h m ( x ) :=
Revised: Similarly, we let ˆ q h m ( x ) :=

Original: [Equation]
Revised: [Equation]

Original: be the KDE of Y and let c Y be the bootstrap bandwidth of (cid:13)(cid:13)
Revised: be the KDE of Y and let c Y be the bootstrap bandwidth of (cid:13)(cid:13)

Original: ˆ q h m − ˆ q ∗ h m (cid:13)(cid:13) ∞ , and then we use ˆ supp( Q )
Revised: ˆ q h m − ˆ q ∗ h m (cid:13)(cid:13) ∞ , and then we use ˆ supp( Q )

Original: =
Revised: =

Original: ˆ q − 1 h m [ c Y , ∞ ) .
Revised: ˆ q − 1 h m [ c Y , ∞ ) .

Original: Using the superlevel set at c X allows to filter out noise whose KDE values are likely to be small.
Revised: Using the superlevel set at c X allows to filter out noise whose KDE values are likely to be small.

Original: For the robust estimates of the precision, we apply the support estimates to the precision of data points, and define the topological precision ( TopP ) as
Revised: For the robust estimates of the precision, we apply the support estimates to the precision of data points, and define the topological precision ( TopP ) as

Original: [Equation]
Revised: [Equation]

Original: And we similarly define the topological recall ( TopR ) as
Revised: And we similarly define the topological recall ( TopR ) as

Original: [Equation]
Revised: [Equation]

Revised: The kernel bandwidths h n and h m are hyperparameters that users need to choose.

Revised: We also provide our guideline to select the optimal bandwidths h n and h m in practice.

Revised: (See Appendix F.3)

Original: 3. B ANDWIDTH ESTIMATION USING BOOTSTRAPPING
Revised: 3. B ANDWIDTH ESTIMATION USING BOOTSTRAPPING

Original: Using the bootstrap bandwidth c X as threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) .
Revised: Using the bootstrap bandwidth c X as threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) .

Original: As we have seen in Section 2, the bootstrap bandwidth c X acts as a threshold for filtering out the topological noise in topological data analysis.
Revised: As we have seen in Section 2, the bootstrap bandwidth c X acts as a threshold for filtering out the topological noise in topological data analysis.

Original: Analogously, using c X as a threshold allows to robustly estimating supp( P ) .
Revised: Analogously, using c X as a threshold allows to robustly estimating supp( P ) .

Original: In particular, when X i is an outlier, its KDE value ˆ p h ( X i ) is likely to be small, and the KDE values at the connected component generated by X i is likely to be small as well.
Revised: When X i is an outlier, its KDE value ˆ p h ( X i ) is likely to be small, and the KDE values at the connected component generated by X i is likely to be small as well.

Original: So those components from outliers are likely to be removed in the estimated support ˆ p − 1 h [ c X , ∞ ) .
Revised: So those components from outliers are likely to be removed in the estimated support ˆ p − 1 h [ c X , ∞ ) .

Original:  Hence, the estimated support denoises topological noisefrom outliers and robustly estimates supp( P ) .
Revised: Higher dimensional homological noises are also removed. Hence, the estimated support denoises topological noise and robustly estimates supp( P ) .

Original: See Section B for more detailed explanation.
Revised: See Appendix B for more detailed explanation.

Original: Now that we are only left with topological features of high confidence, this allows us to draw analogies to confidence intervals in statistical analysis, where the uncertainty of the samples is treated by setting the level of confidence.
Revised: Now that we are only left with topological features of high confidence, this allows us to draw analogies to confidence intervals in statistical analysis, where the uncertainty of the samples is treated by setting the level of confidence.

Original: In the next section, we show that TopP&R not only gives a more reliable evaluation score for generated samples but also has a good theoretical properties.
Revised: In the next section, we show that TopP&R not only gives a more reliable evaluation score for generated samples but also has a good theoretical properties.

Original:  C ONSISTENCY WITH ROBUSTNESS OF T OP P&R
Revised:  C ONSISTENCY WITH ROBUSTNESS OF T OP P&R

Original: The key properties of TopP&R is consistency with robustness.
Revised: The key properties of TopP&R is consistency with robustness.

Original: The consistency ensures that, the precision and the recall we compute from the data approaches the precision and the recall from the distribution as we have more samples.
Revised: The consistency ensures that, the precision and the recall we compute from the data approaches the precision and the recall from the distribution as we have more samples.

Original: The consistency allows to investigate the precision and the recall of the full distributions only with access to finite sampled data.
Revised: The consistency allows to investigate the precision and the recall of the full distributions only with access to finite sampled data.

Original: TopP&R achieves consistency with robustness, that is, the consistency holds with the data possibly corrupted by noise.
Revised: TopP&R achieves consistency with robustness, that is, the consistency holds with the data possibly corrupted by noise.

Original: This is due to the robust estimation of the support with the kernel density estimator with confidence bands.
Revised: This is due to the robust estimation of the support with the kernel density estimator with confidence bands.

Original: This section is devoted to the theoretical analysis of consistency of TopP&R with robustness.
Revised: This section is devoted to the theoretical analysis of consistency of TopP&R with robustness.

Original: We demonstrate the statistical model for the data and the noise.
Revised: We demonstrate the statistical model for the data and the noise.

Original: Let P , Q , X , Y be as in Notation in
Revised: Let P , Q , X , Y be as in Notation in

Original: Section 2, and let X 0 , Y 0 be real data and generated data without noise.
Revised: Section 2, and let X 0 , Y 0 be real data and generated data without noise.

Original: X , Y , X 0 , Y 0 are understood as multisets, i.e. , elements can be repeated.
Revised: X , Y , X 0 , Y 0 are understood as multisets, i.e. , elements can be repeated.

Original: We first assume that the uncorrupted data are IID.
Revised: We first assume that the uncorrupted data are IID.

Original: Assumption 1.
Revised: Assumption 1.

Original: The data X 0 = { X 01 , . . .
Revised: The data X 0 = { X 01 , . . .

Original: , X 0 n } and Y 

Revised: , X 0 n } and Y

Original: = { Y 01 , . . .
Revised: = { Y 01 , . . .

Original: , Y 0 m } are IID from P and Q , respectively.
Revised: , Y 0 m } are IID from P and Q , respectively.

Original: In practice, the data is often corrupted with noise.
Revised: In practice, the data is often corrupted with noise.

Original: We consider the adversarial noise, where some fraction of data are replaced with arbitrary point cloud data.
Revised: We consider the adversarial noise, where some fraction of data are replaced with arbitrary point cloud data.

Original: Assumption 2.
Revised: Assumption 2.

Original: Let { ρ k } k ∈
Revised: Let { ρ k } k ∈

Original: N
Revised: N

Original: be a sequence of nonnegative real numbers.
Revised: be a sequence of nonnegative real numbers.

Original: Then the observed data X and Y satisfies (cid:12)(cid:12) X\X 0 (cid:12)(cid:12) = nρ n and (cid:12)(cid:12) Y\Y 0 (cid:12)(cid:12) =
Revised: Then the observed data X and Y satisfies (cid:12)(cid:12) X\X 0 (cid:12)(cid:12) = nρ n and (cid:12)(cid:12) Y\Y 0 (cid:12)(cid:12) =

Original: mρ m .
Revised: mρ m .

Original: In the adversarial model, we control the level of noise by the fraction ρ , but do not assume other conditions such as IID or boundedness, to make our noise model very general and challenging.
Revised: In the adversarial model, we control the level of noise by the fraction ρ , but do not assume other conditions such as IID or boundedness, to make our noise model very general and challenging.

Original: For distributions, we assume that the order of probability volume decay P ( B ( x, r )) is at least r d .

Original: Assumption 3.

Original: For all x ∈ supp( P ) and y ∈ supp( Q ) ,

Original: [Equation]

Original: Remark 1 .

Original: Assumption 3 is analogous to Assumption 2 of (Kim et al., 2019), but is weaker since the condition is pointwise on each x ∈ R d .

Original: And this condition is much weaker than assuming a density on R d : for example, a distribution supported on a low-dimensional manifold satisfies Assumption 3.

Original: This provides a framework suitable for high dimensional data, since many times high dimensional data lies on a low dimensional structure hence its density on R d cannot exist.

Original: See (Kim et al., 2019) for more detailed discussion.

Original: For  kernel functions, wealso assume weak condition, detailed in Assumption A1  in Section C. Under the data and the noise models, TopP&R achieves consistency with robustness.
Revised: For distributions and kernel functions, we assume weak condition, detailed in Assumption A1 and A in Appendix C. Under the data and the noise models, TopP&R achieves consistency with robustness.

Original: That is, the estimated precision and recall is asymptotically correct with high probability even if up to a portion of 1 / √ n or 1 / √ m are replaced by adversarial noise.
Revised: That is, the estimated precision and recall is asymptotically correct with high probability even if up to a portion of 1 / √ n or 1 / √ m are replaced by adversarial noise.

Original: This is due to the robust estimation of the support with the kernel density estimator with the confidence band of the persistent homology.
Revised: This is due to the robust estimation of the support with the kernel density estimator with the confidence band of the persistent homology.

Original: Proposition 2.
Revised: Proposition 1.

Original: Suppose Assumption 1,2,3,A1 hold.
Revised: Suppose Assumption 1,2,A1,A2 hold.

Original: Suppose h n → 0 , nh n → ∞ , nh − dn ρ 2 n → 0 , and similar relations hold for h m , ρ m .
Revised: Suppose h n → 0 , nh n → ∞ , nh − dn ρ 2 n → 0 , and similar relations hold for h m , ρ m .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Revised: probability

Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 2.

Original: Under the same condition as in Proposition 2, |
Revised: Under the same condition as in Proposition 1,

Original: [Equation]
Revised: [Equation]

Original:  Our theoretical results in Proposition 2 and Theorem 3 are novel and important in several perspetives.
Revised: → 0 , in probability . Our theoretical results in Proposition 1 and Theorem 2 are novel and important in several perspetives.

Original: These results are among the first theoretical guarantees for evaluation metrics for generative models as far as we are aware of.
Revised: These results are among the first theoretical guarantees for evaluation metrics for generative models as far as we are aware of.

Original: Also, as in Remark 1, assumptions are very weak and suitable for high dimensional data.
Revised: Also, as in Remark 3, assumptions are very weak and suitable for high dimensional data.

Original: Also, robustness to adversarial noise is provably guaranteed.
Revised: Also, robustness to adversarial noise is provably guaranteed.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: A good evaluation metric must correctly capture the changes of the underlying data distribution.
Revised: A good evaluation metric must correctly capture the changes of the underlying data distribution.

Original: To examine the performance of evaluation metrics, we carefully select a set of experiments for sanity checks.
Revised: To examine the performance of evaluation metrics, we carefully select a set of experiments for sanity checks.

Original: With toy and real image data, we check 1) how well the metric captures the true trend of underlying data distributions and 2) how well the metric resist perturbations applied to samples. 
Revised: With toy and real image data, we check 1) how well the metric captures the true trend of underlying data distributions and 2) how well the metric resist perturbations applied to samples. The shaded area of the figures denotes the ± 1 standard deviation for ten trials.

Original: 5. S ANITY CHECKS WITH TOY DATA
Revised: 5. S ANITY CHECKS WITH TOY DATA

Original: Following Naeem et al.
Revised: Following Naeem et al.

Original: (2020), we first examine how well the metric reflects the trend of Y moving away from X and whether it is suitable for finding mode-drop phenomena.
Revised: (2020), we first examine how well the metric reflects the trend of Y moving away from X and whether it is suitable for finding mode-drop phenomena.

Original: In addition to these, we newly design several experiments that can highlight TopP&R ’s favorable theoretical properties, such as consistency with robustness, in various scenarios.
Revised: In addition to these, we newly design several experiments that can highlight TopP&R ’s favorable theoretical properties of consistency with robustness in various scenarios.

Original: 5.1. S HIFTING THE GENERATED FEATURE MANIFOLD
Revised: 5.1. S HIFTING THE GENERATED FEATURE MANIFOLD

Original: For this experiment, we generate samples for X ∼ N (0 , I ) and Y ∼ N ( µ 1 , I ) in R 64 where 1 is a vector of ones and I is an identity matrix.
Revised: For this experiment, we generate samples for X ∼ N (0 , I ) and Y ∼ N ( µ 1 , I ) in R 64 where 1 is a vector of ones and I is an identity matrix.

Original: We then examine how each metric responds to shifting
Revised: We then examine how each metric responds to shifting

Original: Y with µ ∈ [ − 1 , 1] while there are outliers at 3 ∈ R 64 for both X and Y (Figure 2).
Revised: Y with µ ∈ [ − 1 , 1] while there are outliers at 3 ∈ R 64 for both X and Y (Figure 2).

Original: Here, we find that both improved P&R and D&C behave pathologically when there are outliers.Since these [Figure] [Figure]
Revised: Here, we find that both improved P&R and D&C behave pathologically when there are outliers.

Original: ten trials. methods are based on the k-nearest neighbor algorithm and ignore the fact that there can be outliers in both real and fake data, they inevitably overestimate the underlying support when there are outliers.
Revised: Since these methods are based on the k-nearest neighbor algorithm and ignore the fact that there can be outliers in both real and fake data, they inevitably overestimate the underlying support when there are outliers.

Original: For example, when X lies between Y and the outlier at y = 3 , Recall returns a high-diversity score, even though the true supports of X and Y are actually far apart.
Revised: For example, when X lies between Y and the outlier at y = 3 , Recall returns a high-diversity score, even though the true supports of X and Y are actually far apart.

Original: In addition, P&R does not reach 1 in high dimensions even when X = Y .
Revised: In addition, P&R does not reach 1 in high dimensions even when X = Y .

Original: Naeem et al.
Revised: Naeem et al.

Original: (2020) circumvented these problems by proposing D&C that always use X (the real data distribution) as a reference point, which in most cases is assumed to have fewer outliers than Y (the fake data distribution).
Revised: (2020) circumvented these problems by proposing D&C that always use X (the real data distribution) as a reference point, which in most cases is assumed to have fewer outliers than Y (the fake data distribution).

Original: However, there is no guarantee that this will be the case in practice.
Revised: However, there is no guarantee that this will be the case in practice.

Original: When there is an outlier in X , D&C also returns an incorrect high-fidelity score at µ > 0 .
Revised: When there is an outlier in X , D&C also returns an incorrect high-fidelity score at µ > 0 .

Original: On the other hand, TopP&R shows a stable trend unaffected by outliers, demonstrating the robustness of our method. 
Revised: On the other hand, TopP&R shows a stable trend unaffected by outliers, demonstrating the robustness of our method. [Figure] [Figure] [Figure]

Original: 5.1. S EQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES
Revised: 5.1. S EQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES

Original: For this experiment, we consider the mixture of Gaussians with seven modes in R 64 .
Revised: For this experiment, we consider the mixture of Gaussians with seven modes in R 64 .

Original: We simulate mode-drop phenomena by gradually dropping all but one mode from the fake distribution Y that is initially identical to X (Figure 3).
Revised: We simulate mode-drop phenomena by gradually dropping all but one mode from the fake distribution Y that is initially identical to X (Figure 3).

Original: As in the illustration of mode-drop experiment, when the number of samples in a particular mode decreases, we kept the number of samples in X constant so that the same amount of decreased samples are supplemented to the first mode which leads fidelity to be fixed to 1.
Revised: As in the illustration of mode-drop experiment, when the number of samples in a particular mode decreases, we kept the number of samples in X constant so that the same amount of decreased samples are supplemented to the first mode which leads fidelity to be fixed to 1.

Original: From the result, we observe that the values of Precision fail to saturate, i.e. , mainly smaller than 1, and the Density fluctuates to a value greater than 1 indicating their instability and unboundedness.
Revised: From the result, we observe that the values of Precision fail to saturate, i.e. , mainly smaller than 1, and the Density fluctuates to a value greater than 1 indicating their instability and unboundedness.

Original: In terms of diversity, Recall does not respond to the simultaneous mode drop, nor does the improved metric Coverage show a fast decay as the reference line.
Revised: In terms of diversity, Recall does not respond to the simultaneous mode drop, nor does the improved metric Coverage show a fast decay as the reference line.

Original: Compared to these methods, TopP performs well, being held at the upperbound of 1 in sequential mode dropping, and
Revised: Compared to these methods, TopP performs well, being held at the upperbound of 1 in sequential mode dropping, and

Original: TopR also decreases closest to the reference line in simultaneous mode drops.
Revised: TopR also decreases closest to the reference line in simultaneous mode drops.

Original: 5.1. T OLERANCE TO N ON -IID PERTURBATIONS
Revised: 5.1. T OLERANCE TO N ON -IID PERTURBATIONS

Original: Robustness to perturbations is another important aspect we should consider when designing a metric.
Revised: Robustness to perturbations is another important aspect we should consider when designing a metric.

Original: Here, we test whether TopP&R behaves stably under two variants of noise cases; 1) scatter noise : replacing X i and Y j with uniformly distributed noise and 2) swap noise : swapping the position between X i and Y j .
Revised: Here, we test whether TopP&R behaves stably under two variants of noise cases; 1) scatter noise : replacing X i and Y j with uniformly distributed noise and 2) swap noise : swapping the position between X i and Y j .

Original: These two cases all correspond to the adversarial noise model of Assumption 2.
Revised: These two cases all correspond to the adversarial noise model of Assumption 2.

Original: We set X ∼ N ( µ = 0 , I
Revised: We set X ∼ N ( µ = 0 , I

Original: ) ∈
Revised: ) ∈

Original: R 64 and Y ∼ N ( µ = 1 , I
Revised: R 64 and Y ∼ N ( µ = 1 , I

Original: ) ∈ R 64 where µ = 1 , and thus an ideal evaluation metric must return zero for both fidelity and diversity scores.
Revised: ) ∈ R 64 where µ = 1 , and thus an ideal evaluation metric must return zero for both fidelity and diversity.

Original: In both cases in Figure 4, we find that P&R and D&C are more sensitive while TopP&R remains relatively stable until the noise [Figure]
Revised: In both cases, we find that P&R and D&C are more sensitive while TopP&R remains relatively stable until the noise ratio reaches 15% of the total data, which is a clear example of the weakness of existing metrics to perturbation (Figure 4).

Original: [Figure]
Revised: [Figure]

Original: ratio reaches 15% of the total data, which is a clear example of the weakness of existing metrics to perturbation.

Original: 5. S ANITY CHECK WITH R EAL DATA
Revised: 5. S ANITY CHECK WITH R EAL DATA

Original: Now that we have verified the metrics on toy data using Gaussians, we test them on real data.
Revised: Now that we have verified the metrics on toy data using Gaussians, we test them on real data.

Original: Just like in the toy experiments, we concentrate on how the metrics behave in extreme situations, such as outliers, mode-drop phenomena, perceptual distortions, and etc.
Revised: Just like in the toy experiments, we concentrate on how the metrics behave in extreme situations, such as outliers, mode-drop phenomena, perceptual distortions, and etc.

Original: For evaluation, ImageNet pretrained 
Revised: We also test the image embedder for evaluation, including pretrained VGG16 (Simonyan & Zisserman, 2014), InceptionV3 (Szegedy et al., 2016), and SwAV (Morozov et al., 2020).

Original: VGG16 with linear random projection to 32 dimension is  used as an image embedding.
Revised: Here, linear random projection to 32 dimension is additionally used for TopP&R .

Original: For more experimental details, please refer to the appendix.
Revised: For more experimental details, please refer to the Appendix F.1.

Original: 5.2. R ESOLVING FIDELITY AND DIVERSITY
Revised: 5.2. R ESOLVING FIDELITY AND DIVERSITY

Original: To test whether TopP&R responds appropriately to the change in the underlying distributions in real scenarios, we test the metric on the generated images of stylegan2 (Karras et al., 2020) using the truncation trick (Karras et al., 2019).
Revised: To test whether TopP&R responds appropriately to the change in the underlying distributions in real scenarios, we test the metric on the generated images of StyleGAN2 (Karras et al., 2020) using the truncation trick (Karras et al., 2019).

Original: As shown in Figure 5, every time the distribution is transformed by ψ , TopP&Rconsistently responds  with bounded scores in [0 , 1] .
Revised: As shown in Figure 5, every time the distribution is transformed by ψ , TopP&R responds well and shows consistent behavior across different embedders with bounded scores in [0 , 1] , which are important virtues as an evaluation metric.

Original: On the other hand, Density gives unbounded scores (fidelity > 1).
Revised: On the other hand, Density gives unbounded scores (fidelity > 1) and shows inconsistent trend depending on the embedder.

Original: Because Density is not capped in value, it is difficult to interpret  and know exactly which value denotes the best performance ( e.g. , in our case, the best performance is when fidelity and diversity = 1).
Revised: Because Density is not capped in value, it is difficult to interpret the score and know exactly which value denotes the best performance ( e.g. , in our case, the best performance is when TopP&R = 1 ).

Revised: Since TopP&R pays more attention to the consistent behavior of a model by examining what the model primarily generates, rather than relying on the entire sample, which contains results by chance, the fact that TopP is kept at 1.0 means that StyleGAN2 produces high-quality images most of the time.

Revised: Thus, this behavior (“ TopP remains constant") does not mean that TopP is inferior to regular precision for checking the trade-off between fidelity and diversity, but rather reveals its property focusing on different perspectives than the others.

Original: 5.2. S EQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES IN CIFAR- 
Revised: 5.2. S EQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES IN CIFAR-

Original: We conduct an additional simultaneous mode drop experiment to verify TopP&R ’s actual sensitiveness on the real data set (CIFAR-10).
Revised: We conduct an additional simultaneous mode drop experiment to verify TopP&R ’s actual sensitiveness on the real data set (CIFAR-10).

Original: The performance of each metric (Figure A1) is measured with the identical data while simultaneously dropping the modes of nine classes of CIFAR10.
Revised: The performance of each metric (Figure A2) is measured with the identical data while simultaneously dropping the modes of nine classes of CIFAR10.

Original: Since the number of the images dropped in each step is identical, the trend of ground truth diversity should linearly decrease.
Revised: Since the number of the images dropped in each step is identical, the trend of ground truth diversity should linearly decrease.

Original: Here, P&R metric captures the simultaneous mode dropping better than D&C because this time random drop of the modes has reduced the area of the estimated fake manifold.
Revised: Here, P&R metric captures the simultaneous mode dropping better than D&C because this time random drop of the modes has reduced the area of the estimated fake manifold.

Original: On the other hand, TopP&R best captures the true trend of decreasing diversity on average, consistent with the toy result in Figure 3.
Revised: On the other hand, TopP&R best captures the true trend of decreasing diversity on average, consistent with the toy result in Figure 3.

Original: [Figure] [Figure]
Revised: In addition, we perform the experiments on a dataset with long-tailed distribution and find that TopP&R captures the trend well even when there are minority sets (Appendix G.2). This again shows the reliability of TopP&R .

Original: 5.2. R OBUSTNESS TO PERTURBATIONS BY OUTLYING FEATURES
Revised: 5.2. R OBUSTNESS TO PERTURBATIONS BY OUTLYING FEATURES

Original: To demonstrate the robustness of our metric against the adversarial noise model of Assumption 2, we test both scatter-noise and swap noise scenarios with real data.
Revised: To demonstrate the robustness of our metric against the adversarial noise model of Assumption 2, we test both scatter-noise and swap noise scenarios with real data.

Original: In the experiment, following
Revised: In the experiment, following

Original: Kynkäänniemi et al.
Revised: Kynkäänniemi et al.

Original: , we first classify inliers and outliers that are generated by StyleGAN (Karras et al., 2019).
Revised: , we first classify inliers and outliers that are generated by StyleGAN (Karras et al., 2019).

Original: For scatter noise we add the outliers to the inliers and for swap noise we swap the real FFHQ images with generated images. 
Revised: For scatter noise we add the outliers to the inliers and for swap noise we swap the real FFHQ images with generated images. Under these specific noise conditions, Precision

Original: Under these specific noise conditions, Precision shows similar or even better robustness than Density (Figure 6).
Revised: [Figure] [Figure] [Figure] shows similar or even better robustness than Density (Figure 6).

Original: On the other hand, Coverage is more robust than Recall .
Revised: On the other hand, Coverage is more robust than Recall .

Original: In both cases, TopP&R shows the best performance, resistant to noise.
Revised: In both cases, TopP&R shows the best performance, resistant to noise.

Original: 5.2. S ENSITIVENESS TO THE NOISE INTENSITY
Revised: 5.2. S ENSITIVENESS TO THE NOISE INTENSITY

Original: One of the advantages of FID (Heusel et al., 2017) is that it is good at estimating the degrees of distortion applied to the images.
Revised: One of the advantages of FID (Heusel et al., 2017) is that it is good at estimating the degrees of distortion applied to the images.

Original: Similarly, we check whether the F1-score based on TopP&R provides a reasonable evaluation according to different noise levels.
Revised: Similarly, we check whether the F1-score based on TopP&R provides a reasonable evaluation according to different noise levels.

Original: As illustrated in Figure 7, X and Y are sets of reference FFHQ features and noisy FFHQ features, respectively.
Revised: As illustrated in Figure 7, X and Y are sets of reference FFHQ features and noisy FFHQ features, respectively.

Original: The experimental results show that TopP&R actually reflects well the different degrees of distortion added to the images.
Revised: The experimental results show that TopP&R actually reflects well the different degrees of distortion added to the images.

Original: 5.2. R ANKING BETWEEN GENERATIVE MODELS
Revised: 5.2. R ANKING BETWEEN GENERATIVE MODELS

Original: One of the major caveats with two-score metrics is that they make it difficult to rank between different models; e.g. , which model is better?
Revised: One of the major caveats with two-score metrics is that they make it difficult to rank between different models; e.g. , which model is better?

Original: High fidelity with low diversity or low fidelity with high diversity?
Revised: High fidelity with low diversity or low fidelity with high diversity?

Original: In the case of traditional precision and Recall, this problem could be solved by using F1-score, which is the harmonic mean of fidelity and diversity.
Revised: In the case of traditional precision and Recall, this problem could be solved by using F1-score, which is the harmonic mean of fidelity and diversity.

Original: However, unlike the traditional ones, the F1-score based on P&R or D&C does not provide a reliable or stable score due to their inherent instability and unboundedness.
Revised: However, unlike the traditional ones, the F1-score based on P&R or D&C does not provide a reliable or stable score due to their inherent instability and unboundedness.

Original: Thanks to its stability and robustness to various perturbations, we find that the TopP&R -based F1 score offers consistent ranking with FID  (Table 1) 1 .
Revised: Thanks to its stability and robustness to various perturbations, we find that the TopP&R -based F1 score offers consistent ranking with FID under various embedding networks (Table 1) 1 .

Original: [Table]
Revised: To quantitatively compare between the similarity of rankings across varying embedders by different metrics, we have computed Hamming Distance (HD) (Appendix F.4) where lower HD indicates more similarity. TopP&R , P&R , and D&C have HDs of 1.33, 2.66, and 3.0, respectively. From this, TopP&R provides the most consistent ranking across varying embedders (consistent to Section 5.2.1).

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Persistent homology and deep learning.
Revised: Persistent homology and deep learning.

Original: Topology shows various potentials in the field of deep learning by introducing a new perspective on the support estimation, a new distance function robust to the noisy information, and a technique for GAN evaluation.
Revised: Topology shows various potentials in the field of deep learning by introducing a new perspective on the support estimation, a new distance function robust to the noisy information, and a technique for GAN evaluation.

Original: Chen et al.
Revised: Chen et al.

Original: (2017) introduces a method for  approximating the support of a distribution using general density estimator and the Hausdorff distance and a new visualization method for support.
Revised: (2017) introduces a method for [Table] approximating the support of a distribution using general density estimator and the Hausdorff distance and a new visualization method for support.

Original: Chazal et al.
Revised: Chazal et al.

Original: (2011) proposes distance-to-measure, a robust Wasserstein distance function for perturbation, as an alternative to the characteristic that existing distance functions are not robust to outliers.
Revised: (2011) proposes distance-to-measure, a robust Wasserstein distance function for perturbation, as an alternative to the characteristic that existing distance functions are not robust to outliers.

Original: For the evaluation, one of the recent metric called MTop-Divergence (Barannikov et al., 2021) uses the summation (or in another word statistics) of the life-length of homology to score which manifold is containing more important topological signals.
Revised: For the evaluation, one of the recent metric called MTop-Divergence (Barannikov et al., 2021) uses the summation (or in another word statistics) of the life-length of homology to score which manifold is containing more important topological signals.

Original: While M-Top-Divergence directly use persistent homology to score the deep-learning models, we employ topology to estimate a robust and stable manifold.
Revised: While MTop-Divergence directly use persistent homology to score the deep-learning models, we employ topology to estimate a robust and stable manifold.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: Various evaluation metrics for generative models have been recently proposed (Salimans et al., 2016;
Revised: Various evaluation metrics for generative models have been recently proposed (Salimans et al., 2016;

Original: Heusel et al., 2017; Sajjadi et al., 2018; Kynkäänniemi et al., 2019; Naeem et al., 2020; Borji, 2022).
Revised: Heusel et al., 2017; Sajjadi et al., 2018; Kynkäänniemi et al., 2019; Naeem et al., 2020; Borji, 2022).

Original: One of the earliest methods is Inception Score (IS) (Szegedy et al., 2016), which measures the divergence of generated samples on the InceptionV3 embedding space.
Revised: One of the earliest methods is Inception Score (IS) (Szegedy et al., 2016), which measures the divergence of generated samples on the InceptionV3 embedding space.

Original: However, IS fails to capture the simultaneous mode drop and only considers the population distribution.
Revised: However, IS fails to capture the simultaneous mode drop and only considers the population distribution.

Original: Fréchet Inception Distance (FID) (Heusel et al., 2017) measures the difference in the means and variances of the real and fake features.
Revised: Fréchet Inception Distance (FID) (Heusel et al., 2017) measures the difference in the means and variances of the real and fake features.

Original: Since FID assumes the multi-Gaussian distribution of the features, if the true feature distribution is not normally distributed, the estimation becomes highly unreliable.
Revised: Since FID assumes the multi-Gaussian distribution of the features, if the true feature distribution is not normally distributed, the estimation becomes highly unreliable.

Original: Unlike IS and FID, which give a single score, some metrics separate the score into two components, the fidelity and diversity Sajjadi et al.
Revised: Unlike IS and FID, which give a single score, some metrics separate the score into two components, the fidelity and diversity Sajjadi et al.

Original: ; Kynkäänniemi et al.
Revised: ; Kynkäänniemi et al.

Original: ; Naeem et al.
Revised: ; Naeem et al.

Original: While Topological Precision and Recall ( TopP&R ) falls into this category, unlike the others, it does not assume strong regularity conditions.
Revised: While Topological Precision and Recall ( TopP&R ) falls into this category, unlike the others, it does not assume strong regularity conditions.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Recently, many works have been proposed to score the fidelity and diversity of generative models.
Revised: Recently, many works have been proposed to score the fidelity and diversity of generative models.

Original: However, none of them has focused on an accurate estimation of supports even though this is one of the key components in the evaluation pipeline.
Revised: However, none of them has focused on an accurate estimation of supports even though this is one of the key components in the evaluation pipeline.

Original: In this paper, we proposed topological precision and recall ( TopP&R ) that provides a systematical fix for robustly estimating the manifold by employing topological and statistical ideas.
Revised: In this paper, we proposed topological precision and recall ( TopP&R ) that provides a systematical fix for robustly estimating the manifold by employing topological and statistical ideas.

Original: Our theoretical and experimental results showed that TopP&R serves as a robust and reliable evaluation metric under various  noisy conditions, including mode collapse, outliers, and Non-IID perturbations.
Revised: Our theoretical and experimental results showed that TopP&R serves as a robust and reliable evaluation metric under various embeddings and noisy conditions, including mode collapse, outliers, and Non-IID perturbations.

Original: A PPENDIX A M ORE B ACKGROUND ON T OPOLOGICAL D ATA A NALYSIS
Revised: A PPENDIX A M ORE B ACKGROUND ON T OPOLOGICAL D ATA A NALYSIS

Original: Topological data analysis (TDA) (Carlsson, 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data.
Revised: Topological data analysis (TDA) (Carlsson, 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data.

Original: A key object in TDA is persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.
Revised: A key object in TDA is persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.

Original: Filtration.
Revised: Filtration.

Original: A filtration is a collection of subspaces approximating the data points at different resolutions, formally F = {F δ ⊂ R d } δ ∈ R such that δ 1 ≤ δ 2 implies that F δ 1 ⊂ F δ 2 .
Revised: A filtration is a collection of subspaces approximating the data points at different resolutions, formally F = {F δ ⊂ R d } δ ∈ R such that δ 1 ≤ δ 2 implies that F δ 1 ⊂ F δ 2 .

Original: Typically, a filtration is defined through a function f related to data.
Revised: Typically, a filtration is defined through a function f related to data.

Original: Given a function f : R d → R , we consider its sublevel filtration { f − 1 ( −∞ , δ ] } δ ∈ R or a superlevel filtration { f − 1 [ δ, ∞ ) } δ ∈ R .
Revised: Given a function f : R d → R , we consider its sublevel filtration { f − 1 ( −∞ , δ ] } δ ∈ R or a superlevel filtration { f − 1 [ δ, ∞ ) } δ ∈ R .

Original: Persistent homology.
Revised: Persistent homology.

Original: Persistent homology is a multiscale approach to represent the topological features, and is represented in the persistence diagram.
Revised: Persistent homology is a multiscale approach to represent the topological features, and is represented in the persistence diagram.

Original: For a filtration F and for each nonnegative k , we track when k -dimensional homological features (e.g., 0 -dimension: connected component, 1 -dimension: loop, 2 -dimension: cavity, . . .
Revised: For a filtration F and for each nonnegative k , we track when k -dimensional homological features (e.g., 0 -dimension: connected component, 1 -dimension: loop, 2 -dimension: cavity, . . .

Original: ) appear and disappear in the filtration.
Revised: ) appear and disappear in the filtration.

Original: As increasing or decreasing δ in the filtration {F δ } , if a homological feature appears at F b and disappears at F d , then we say that it is born at b and dies at d .
Revised: As increasing or decreasing δ in the filtration {F δ } , if a homological feature appears at F b and disappears at F d , then we say that it is born at b and dies at d .

Original: By considering these pairs { ( b, d ) } as points in the plane ( R ∪ {±∞} ) 2 , we obtain a persistence diagram .
Revised: By considering these pairs { ( b, d ) } as points in the plane ( R ∪ {±∞} ) 2 , we obtain a persistence diagram .

Original: From this, a homological feature with a longer life length, d − b , can be treated as a significant feature in the data set, and a homological feature with a shorter life length as a topological noise, which lies near the diagonal line { ( δ, δ ) : δ ∈ R } .
Revised: From this, a homological feature with a longer life length, d − b , can be treated as a significant feature in the data set, and a homological feature with a shorter life length as a topological noise, which lies near the diagonal line { ( δ, δ ) : δ ∈ R } .

Original: As discussed above, a homological feature with a long life-length is an important information in topology while the homology with short life-length can be treated as a non-significant information or noise.
Revised: As discussed above, a homological feature with a long life-length is an important information in topology while the homology with short life-length can be treated as a non-significant information or noise.

Original: The confidence band estimator provides the confidence set from the features that only includes topologically and statistically significant (statistically considered as elements in the population set) under a certain level of confidence.
Revised: The confidence band estimator provides the confidence set from the features that only includes topologically and statistically significant (statistically considered as elements in the population set) under a certain level of confidence.

Original: One way of constructing the confidence set uses the superlevel filtration of kernel density estimator and the bootstrap confidence band.
Revised: One way of constructing the confidence set uses the superlevel filtration of kernel density estimator and the bootstrap confidence band.

Original: Let X
Revised: Let X

Original: = { X 1 , X 2 , ..., X n } as given points cloud, then the probability for the distribution of points can be estimated via KDE defined as following: ˆ p h ( x ) :
Revised: = { X 1 , X 2 , ..., X n } as given points cloud, then the probability for the distribution of points can be estimated via KDE defined as following: ˆ p h ( x ) :

Original: = 1 nh d (cid:80) ni =1 K ( x − X
Revised: = 1 nh d (cid:80) ni =1 K ( x − X

Original: i
Revised: i

Original: h ) where h is the bandwidth and d as a dimension of the space.
Revised: h ) where h is the bandwidth and d as a dimension of the space.

Original: We derive estimated likelihood of X with KDE and likehood of ˜ p with using bootstrapped samples X ∗ .
Revised: We derive estimated likelihood of X with KDE and likehood of ˜ p with using bootstrapped samples X ∗ .

Original: Now, given the significance level α and h > 0 , let confidence band
Revised: Now, given the significance level α and h > 0 , let confidence band

Original: q X be bootstrap bandwidth of a Gaussian Empirical Process (van der Vaart, 2000; Kosorok, 2008), √ n ||
Revised: q X be bootstrap bandwidth of a Gaussian Empirical Process (van der Vaart, 2000; Kosorok, 2008), √ n ||

Original: ˆ p h − ˆ p ∗ h || ∞ .
Revised: ˆ p h − ˆ p ∗ h || ∞ .

Original: Then it satisfies P ( √ n || ˆ p
Revised: Then it satisfies P ( √ n || ˆ p

Original: h − p h
Revised: h − p h

Original: || ∞ <
Revised: || ∞ <

Original: q X ) ≥ 1 − α , as in Proposition 4 in Section C.
Revised: q X ) ≥ 1 − α , as in Proposition 4 in Section C.

Original: Then the ball of persistent homology centered at ˆ P h and radius c X = q X / √ n in the bottleneck distance d B is a valid confidence set as lim inf n →∞ P (cid:16) P ∈ B d B ( ˆ P h , c X ) (cid:17) ≥ 1 − α .
Revised: Then the ball of persistent homology centered at ˆ P h and radius c X = q X / √ n in the bottleneck distance d B is a valid confidence set as lim inf n →∞ P (cid:16) P ∈ B d B ( ˆ P h , c X ) (cid:17) ≥ 1 − α .

Original: This confidence set has further interpretation that in the persistence diagram, homological features that are above twice the radius 2 c X from the diagonal are simultaneously statistically significant.
Revised: This confidence set has further interpretation that in the persistence diagram, homological features that are above twice the radius 2 c X from the diagonal are simultaneously statistically significant.

Original: B D ENOISING TOPOLOGICAL FEATURES FROM OUTLIERS
Revised: B D ENOISING TOPOLOGICAL FEATURES FROM OUTLIERS

Original: Using the bootstrap bandwidth c X as threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) .
Revised: Using the bootstrap bandwidth c X as threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) .

Revised: When the level set ˆ p − 1 h [ c X , ∞ ) is used, the homology of ˆ p − 1 h [ c X , ∞ ) consists of homological features whose (birth) ≥ c X and ( death ) ≤ c X , which are the homological features in skyblue area in Figure A1.

Revised: In this

Original: When the level set ˆ p − 1 h [ c X , ∞ ) is used, the homology of ˆ p − 1 h [ c X , ∞ ) consists of homological features whose (birth) ≥ c X and ( death ) ≤ c X , which are the homological features in skyblue area in  Figure 1.
Revised: example, we conisder three types of homological noise, though there can be many more corresponding to different homological dimensions. • There can be a 0 -dimensional homological noise of (birth) < c X and (death) < c X , which is the red point in the persistence diagram of Figure A1.

Original: Particularly, the  homological features generated by outliers have their (birth) <
Revised: This noise corresponds to the orange connected component on the left. As in the figure, this type of homological noise usually corresponds to outliers. • There can be a 0 -dimensional homological noise of (birth) > c X and (death) >

Revised: c X , which is the green point in the persistence diagram of Figure A1.

Revised: This noise corresponds to the connected component surrounded by green line on the left.

Revised: As in the figure, this type of homological noise lies within the estimated support, not like the other two.

Revised: Data and KDE levels

Original: c X simultaneously with high probability, so those homological noise are removed in the estimated support ˆ p − 1 h [ c X , ∞ ) .
Revised: [Figure] • There can be a 1 -dimensional homological noise of (birth) < c X and (death) < c X , which is the purple point in the persistence diagram of Figure A1.

Revised: This noise corresponds to the purple loop on the left.

Revised: These homological noises satisfy either their (birth) <

Revised: c X and (death) <

Revised: c X or their (birth) >

Revised: c X and (death) >

Revised: c X simultaneously with high probability, so those homological noises are removed in the estimated support ˆ p − 1 h [ c X , ∞ ) , which is the blue area in the left and the skyblue area in the right in Figure A1.

Revised: We would like to further emphasize that homological noises are not restricted to 0 -dimension lying outside the estimated support (red point in the persistence diagram of Figure A1).

Revised: 0dimensional homological noise inside the estimated support (green point in the persistence diagram of Figure A1), 1 -dimensional homological noise can also arise, and the bootstrap bandwidth c X allows to simultaneously filter them.

Original: C A SSUMPTIONS ON DISTRIBUTIONS AND KERNELS
Revised: C A SSUMPTIONS ON DISTRIBUTIONS AND KERNELS

Revised: For distributions, we assume that the order of probability volume decay P ( B ( x, r )) is at least r d .

Original: Assumption A1.
Revised: Assumption A1.

Original: Let K : R d → R be a nonnegative function with ∥ K ∥ 1 = 1 , ∥ K ∥ ∞ , ∥ K
Revised: For all x ∈ supp( P ) and y ∈ supp( Q ) , [Equation] Remark 3 . Assumption A1 is analogous to Assumption 2 of Kim et al.

Revised: but is weaker since the condition is pointwise on each x ∈ R d .

Revised: And this condition is much weaker than assuming a density on R d : for example, a distribution supported on a low-dimensional manifold satisfies Assumption A1.

Revised: This provides a framework suitable for high dimensional data, since many times high dimensional data lies on a low dimensional structure hence its density on R d cannot exist.

Revised: See Kim et al.

Revised: for more detailed discussion.

Revised: For kernel functions, we assume the following regularity conditions:

Revised: Assumption A2.

Revised: Let K : R d → R be a nonnegative function with ∥ K ∥ 1 = 1 , ∥ K ∥ ∞ , ∥ K

Original: ∥
Revised: ∥

Original: 2 < ∞ , and satisfy the following
Revised: 2 < ∞ , and satisfy the following:

Original: : (1) K (0) > 0 .
Revised: (1) K (0) > 0 .

Original: (2) K has a compact support.
Revised: (2) K has a compact support.

Original: (3) K is Lipschitz continuous and of second order.
Revised: (3) K is Lipschitz continuous and of second order.

Original: Assumption A1, allows to build a valid bootstrap confidence band for kernel density estimator (KDE).
Revised: Assumption A2, allows to build a valid bootstrap confidence band for kernel density estimator (KDE).

Original: See Theorem 12 of (Fasy et al., 2014) or Theorem 3.4 of (Neumann, 1998)
Revised: See Theorem 12 of (Fasy et al., 2014) or Theorem 3.4 of (Neumann, 1998)

Original: Proposition 4 (Theorem 3.4 of (Neumann, 1998)) .
Revised: Proposition 4 (Theorem 3.4 of (Neumann, 1998)) .

Original: Let X
Revised: Let X

Original: =
Revised: =

Original: X 1 , . .
Revised: X 1 , . .

Original: X n } be IID from a distribution
Revised: X n } be IID from a distribution

Original: P .
Revised: P .

Original: For h > 0 , let ˆ p h , ˆ p ∗ h be kernel density estimator for X and its bootstrap X ∗ , respectrively, and for α ∈ (0 , 1) , let c
Revised: For h > 0 , let ˆ p h , ˆ p ∗ h be kernel density estimator for X and its bootstrap X ∗ , respectrively, and for α ∈ (0 , 1) , let c

Original: X
Revised: X

Original: be the α bootstrap quantile from √ nh d ∥ ˆ p h − ˆ p ∗ h ∥ ∞ .
Revised: be the α bootstrap quantile from √ nh d ∥ ˆ p h − ˆ p ∗ h ∥ ∞ .

Original: For h n → 0 ,
Revised: For h n → 0 ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 3, A1 ensures that, when the bandwidth h n → 0 , average KDEs are bounded away from 0 .
Revised: Assumption A1, A2 ensures that, when the bandwidth h n → 0 , average KDEs are bounded away from 0 .

Original: Lemma 5.
Revised: Lemma 5.

Original: Let P be a distribution satisfying Assumption 3.
Revised: Let P be a distribution satisfying Assumption A1.

Original: Suppose K is a nonnegative function satisfying K (0) > 0 and continuous at 0 .
Revised: Suppose K is a nonnegative function satisfying K (0) > 0 and continuous at 0 .

Original: Suppose { h n } n ∈ N with h n ≥ 0 and h n → 0 is given.
Revised: Suppose { h n } n ∈ N with h n ≥ 0 and h n → 0 is given.

Original: Then for all x ∈ supp( P ) , lim inf [Equation]
Revised: Then for all x ∈ supp( P ) , lim inf n p h n ( x ) > 0 .

Original: Proof.
Revised: Proof.

Original: Since K (0) > 0 and K is continuous at 0 , there is r 0 > 0 such that for all y ∈ B (0 , r 0 ) ,
Revised: Since K (0) > 0 and K is continuous at 0 , there is r 0 > 0 such that for all y ∈ B (0 , r 0 ) ,

Original: K ( y ) ≥ 12 K (0) > 0 .
Revised: K ( y ) ≥ 12 K (0) > 0 .

Original: And hence
Revised: And hence

Original: [Equation]
Revised: [Equation]

Original:  d P ( B ( x, r 0 h )) .
Revised: (cid:19) dP ( y ) ≥ K (0) 2 h d P ( B ( x, r 0 h )) .

Original: Hence as h n → 0 , lim inf n p h [Equation] x 
Revised: Hence as h n → 0 , lim inf n p h n ( x ) > 0 .

Original: [Equation]

Original: >

Original: D D ETAILS AND P ROOFS FOR S ECTION
Revised: D D ETAILS AND P ROOFS FOR S ECTION

Original: Let ˜ p
Revised: Let ˜ p

Original: h be the KDE on X 0 .
Revised: h be the KDE on X 0 .

Original: For a finite set X , we use the notation c X ,α for α -bootstrap quantile satisfying P (cid:0)(cid:13)(cid:13)
Revised: For a finite set X , we use the notation c X ,α for α -bootstrap quantile satisfying P (cid:0) (cid:13) (cid:13)

Original: ˆ p X ,h n − ˆ p X b ,h n (cid:13)(cid:13) ∞ >
Revised: ˆ p X ,h n − ˆ p X b ,h n (cid:13)(cid:13) ∞ >

Original: c X ,α |X (cid:1)
Revised: c X ,α |X (cid:1)

Original: = 1 − α , where X b is the bootstrap sample from X .
Revised: = 1 − α , where X b is the bootstrap sample from X .

Original: For a distribution P , we use the notation c P,α for α -quantile satisfying P ( ∥ ˆ p h n − p h n ∥ ∞ > c P,α )
Revised: For a distribution P , we use the notation c P,α for α -quantile satisfying P ( ∥ ˆ p h n − p h n ∥ ∞ > c P,α )

Original: = 1 − α , where ˆ p h n is kernel density estimator of IID samples from P .
Revised: = 1 − α , where ˆ p h n is kernel density estimator of IID samples from P .

Original: Hence when X is not IID samples from P , the relation of Proposition 4 may not hold.
Revised: Hence when X is not IID samples from P , the relation of Proposition 4 may not hold.

Original: Lemma 6.
Revised: Lemma 6.

Original:  Under Assumption 1, 2 and A1,
Revised: (1) Under Assumption 1, 2 and A2,

Original: [Equation]
Revised: [Equation]

Original: (2) Under Assumption 1, 2 and A1,
Revised: (2) Under Assumption 1, 2 and A2,

Original: [Equation]
Revised: [Equation]

Original: (3) Suppose Assumption 1, 2,A1 hold, and suppose nh − dn ρ 2 n → 0 .
Revised: (3) Suppose Assumption 1, 2,A2 hold, and suppose nh − dn ρ 2 n → 0 .

Original: Then with probability1 −
Revised: Then with probability

Original: [Equation]
Revised: [Equation]

Original:  2 δ,
Revised: − 2 δ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, note that
Revised: First, note that

Original: [Equation]
Revised: [Equation]

Original: Then under Assumption A1,
Revised: Then under Assumption A2,

Original: [Equation]
Revised: [Equation]

Original: Then from Assumption 2, (cid:80) ni
Revised: Then from Assumption 2, (cid:80) ni

Original: =

Revised: =

Original: I (cid:0) X i ̸
Revised: I (cid:0) X i ̸

Original: =
Revised: =

Original: X 

Revised: X

Original: i (cid:1) ≤ nρ n , and hence
Revised: i (cid:1) ≤ nρ n , and hence

Original: [Equation]
Revised: [Equation]

Original: Let X b , X 0 b be bootstrapped samples of X , X 0 with the same sampling with replacement process.
Revised: Let X b , X 0 b be bootstrapped samples of X , X 0 with the same sampling with replacement process.

Original: Let ˆ p bh , ˜ p
Revised: Let ˆ p bh , ˜ p

Original: bh be KDE of X b and X 0 b , respectively.
Revised: bh be KDE of X b and X 0 b , respectively.

Original: And, note that
Revised: And, note that

Original: [Equation]
Revised: [Equation]

Original: Let L b be the number of elements where X b and X 0 b differ, i.e., L b = (cid:12)(cid:12)
Revised: Let L b be the number of elements where X b and X 0 b differ, i.e., L b = (cid:12)(cid:12)

Original: X b \X 0 b (cid:12)(cid:12) = (cid:12)(cid:12)
Revised: X b \X 0 b (cid:12)(cid:12) = (cid:12)(cid:12)

Original: X 

Revised: X

Original: b \X b (cid:12)(cid:12) , then
Revised: b \X b (cid:12)(cid:12) , then

Original: L b ∼ Binomial( n, ρ n ) , and(cid:13)(cid:13)
Revised: L b ∼ Binomial( n, ρ n ) , and

Original: [Equation]
Revised: [Equation]

Original: And hence,
Revised: And hence,

Original: [Equation]
Revised: [Equation]

Original: Then by using subgaussian tail bound, with probability 1 − δ ,
Revised: Then by using subgaussian tail bound, with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence this implies
Revised: Hence this implies

Original: [Equation]
Revised: [Equation]

Original: Since X 0 is IID samples from P , with probability 1 − α − 2 δ ,
Revised: Since X 0 is IID samples from P , with probability 1 − α − 2 δ ,

Original: [Equation]
Revised: [Equation]

Original: {+X 0 ,α +
+} δ + O (cid:18)nh d (cid:19) .
Revised: {+X 0 ,α +
+} δ + O (cid:18)nh d (cid:19) .

Original: Now, note that c X 0 ,α = Θ (cid:18)(cid:113) log(1 /α )
Revised: Now, note that c X 0 ,α = Θ (cid:18)(cid:113) log(1 /α )

Original: nh d (cid:19) , and hence
Revised: nh d (cid:19) , and hence

Original: [Equation]
Revised: [Equation]

Original: Then under Assumption 2, since nh − dn ρ 2 n = o (1) and h − dn ρ n = o (
Revised: Then under Assumption 2, since nh − dn ρ 2 n = o (1) and h − dn ρ n = o (

Original: [Equation]
Revised: [Equation]

Original: ≤ c
Revised: ≤ c

Original: [Equation]
Revised: [Equation]

Original: α −
Revised: α −

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: Corollary 7.
Revised: Corollary 7.

Original: Suppose Assumption 1, 2, A1 hold.
Revised: Suppose Assumption 1, 2, A2 hold.

Original: (1) Suppose nh − dn ρ 2 n → 0 .
Revised: (1) Suppose nh − dn ρ 2 n → 0 .

Original: Then with probability 1 − α − 2 δ , p − 1 h n [2 c P,α − 2 δ , ∞ ) ⊂
Revised: Then with probability 1 − α − 2 δ , p − 1 h n [2 c P,α − 2 δ , ∞ ) ⊂

Original: ˆ p − 1 h n [ c X ,α , ∞ ) ⊂ supp( P h n ) .
Revised: ˆ p − 1 h n [ c X ,α , ∞ ) ⊂ supp( P h n ) .

Original: (2) Suppose mh − dm ρ 2 m → 0 .
Revised: (2) Suppose mh − dm ρ 2 m → 0 .

Original: Then with probability 1 − α − 2 δ , q
Revised: Then with probability 1 − α − 2 δ , q

Original: [Equation]
Revised: [Equation]

Original: 1 h m
Revised: 1 h m

Original: [Equation]
Revised: [Equation]

Original: 2 c Q,α − 2 δ , ∞ ) ⊂
Revised: 2 c Q,α − 2 δ , ∞ ) ⊂

Original: ˆ q − 1 h m [ c Y ,α , ∞ )
Revised: ˆ q − 1 h m [ c Y ,α , ∞ )

Original: ⊂ supp( Q h m ) .
Revised: ⊂ supp( Q h m ) .

Original: Proof.
Revised: Proof.

Original: (1) From Lemma 6, ∥ ˆ p h − p h
Revised: (1) From Lemma 6, ∥ ˆ p h − p h

Original: ∥ <
Revised: ∥ <

Original: c X ,α ≤ c P,α − 2 δ .
Revised: c X ,α ≤ c P,α − 2 δ .

Original: This implies p − 1 h n [2 c P,α − 2 δ , ∞ ) ⊂ ˆ p − 1 h n [ c X ,α , ∞ )
Revised: This implies p − 1 h n [2 c P,α − 2 δ , ∞ ) ⊂ ˆ p − 1 h n [ c X ,α , ∞ )

Original: ⊂ supp( P h n ) .
Revised: ⊂ supp( P h n ) .

Original: (2) can be proven similarly to (1).
Revised: (2) can be proven similarly to (1).

Original: Claim 8 .
Revised: Claim 8 .

Original: For a nonnegative measure µ and sets A, B, C, D, µ ( A ∩ B ) − µ ( C ∩ D ) ≤ µ ( A \ C ) + µ ( B \ D ) .
Revised: For a nonnegative measure µ and sets A, B, C, D, µ ( A ∩ B ) − µ ( C ∩ D ) ≤ µ ( A \ C ) + µ ( B \ D ) .

Original: Proof.
Revised: Proof.

Original: µ(
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: ) − µ ( C ∩ D ) ≤ µ (( A ∩ B ) \ ( C ∩ D )) = µ (( A ∩ B ) ∩ ( C ∁ ∪ D ∁ ))
Revised: ) − µ ( C ∩ D ) ≤ µ (( A ∩ B ) \ ( C ∩ D )) = µ (( A ∩ B ) ∩ ( C ∁ ∪ D ∁ ))

Original: = µ (( A ∩ B ) ∩ C ∁ ) ∪ ( A ∩ B ) ∩ D ∁ )) ≤ µ (( A ∩ B ) \ C ) + µ ( A ∩ B ) \ D ) ≤ µ ( A \ C ) + µ ( B \ D ) .
Revised: = µ (( A ∩ B ) ∩ C ∁ ) ∪ ( A ∩ B ) ∩ D ∁ )) ≤ µ (( A ∩ B ) \ C ) + µ ( A ∩ B ) \ D ) ≤ µ ( A \ C ) + µ ( B \ D ) .

Original: From here, let P n and Q m be the empirical measures on X and Y ,
Revised: From here, let P n and Q m be the empirical measures on X and Y ,

Original: respectively, i.e., P n = 1 n (cid:80) ni
Revised: respectively, i.e., P n = 1 n (cid:80) ni

Original: =

Revised: =

Original: δ X i and Q m = 1 m (cid:80) mj
Revised: δ X i and Q m = 1 m (cid:80) mj

Original: =1 δ Y j .
Revised: =1 δ Y j .

Original: Lemma 9.
Revised: Lemma 9.

Original: Suppose Assumption 1, 2 hold.
Revised: Suppose Assumption 1, 2 hold.

Original: (1) Let A ⊂ R d .
Revised: (1) Let A ⊂ R d .

Original: Then with probability 1 − δ , | P n − P | ( A ) =
Revised: Then with probability 1 − δ , | P n − P | ( A ) =

Original: o (1) .
Revised: o (1) .

Original: (2) Let A n ⊂ R d be a sequence of set satisfying A n → ∅ , i.e., lim sup n A n = ∅ .
Revised: (2) Let A n ⊂ R d be a sequence of set satisfying A n → ∅ , i.e., lim sup n A n = ∅ .

Original: Then P n ( A n ) → 0 in probability .
Revised: Then P n ( A n ) → 0 in probability .

Original: Proof.
Revised: Proof.

Original: Since P ( A n ) ≤ P ( (cid:83) ni =1 A i ) and (cid:83)
Revised: Since P ( A n ) ≤ P ( (cid:83) ni =1 A i ) and (cid:83)

Original: ni
Revised: ni

Original: =

Revised: =

Original: A i → ∅ as well, we can assume that A n ↓ ∅ , i.e.,
Revised: A i → ∅ as well, we can assume that A n ↓ ∅ , i.e.,

Original: A n ⊃ A n +1 for all n and (cid:84) ∞ n =1 A n = ∅ .
Revised: A n ⊃ A n +1 for all n and (cid:84) ∞ n =1 A n = ∅ .

Original: Let P 0 n be the empirical measure on X 0 , i.e., P 0 n = 1 n (cid:80) ni
Revised: Let P 0 n be the empirical measure on X 0 , i.e., P 0 n = 1 n (cid:80) ni

Original: =1 δ X i .
Revised: =1 δ X i .

Original: By using subgaussian tail bound, with probability 1 − δ , (cid:12)(cid:12) P
Revised: By using subgaussian tail bound, with probability 1 − δ , (cid:12)(cid:12) P

Original: [Equation]
Revised: [Equation]

Original: n − P (cid:12)(cid:12) ( A ) = O (cid:32)(cid:114) log(1 /δ )
Revised: n − P (cid:12)(cid:12) ( A ) = O (cid:32)(cid:114) log(1 /δ )

Original: n (cid:33) .
Revised: n (cid:33) .

Original: And (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12)
Revised: And (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12)

Original: ( A ) is expanded as (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12) ( A ) = 1 n n (cid:88) i = (cid:12)(cid:12)
Revised: ( A ) is expanded as (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12) ( A ) = 1 n n (cid:88) i = (cid:12)(cid:12)

Original: I ( X
Revised: I ( X

Original: i ∈
Revised: i ∈

Original: A )
Revised: A )

Original: − I ( X 0 i ∈ A ) (cid:12)(cid:12) .
Revised: − I ( X 0 i ∈ A ) (cid:12)(cid:12) .

Original: Under Assumption 2 , (cid:80) ni =1 I (cid:0) X i ̸
Revised: Under Assumption 2 , (cid:80) ni =1 I (cid:0) X i ̸

Original: =
Revised: =

Original: X 

Revised: X

Original: i (cid:1) ≤ nρ n , and hence (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12) ( A ) = 1 n n (cid:88) i = (cid:12)(cid:12) I
Revised: i (cid:1) ≤ nρ n , and hence (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12) ( A ) = 1 n n (cid:88) i = (cid:12)(cid:12) I

Original: ( X i ∈ A ) − I
Revised: ( X i ∈ A ) − I

Original: ( X 0 i ∈ A ) (cid:12)(cid:12) ≤n n (cid:88) i =
Revised: ( X 0 i ∈ A ) (cid:12)(cid:12) ≤n n (cid:88) i =

Original: I (cid:0) X i ̸
Revised: I (cid:0) X i ̸

Original: =
Revised: =

Original: X 0 i (cid:1) =
Revised: X 0 i (cid:1) =

Original: ρ n .
Revised: ρ n .

Original: Therefore, with probability 1 − δ , | P n − P | ( A ) ≤ O (cid:32) ρ n +
Revised: Therefore, with probability 1 − δ , | P n − P | ( A ) ≤ O (cid:32) ρ n +

Original: [Equation]
Revised: [Equation]

Original: (2) Note that P n ( A n ) can be bounded as
Revised: (2) Note that P n ( A n ) can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Fix δ > 0 and ϵ > 0 .
Revised: Fix δ > 0 and ϵ > 0 .

Original: Since lim n P ( A n ) = 0 , we can choose N 1 such that P
Revised: Since lim n P ( A n ) = 0 , we can choose N 1 such that P

Original: ( A N 1 )
Revised: ( A N 1 )

Original: <
Revised: <

Original: ϵ , and we can choose N 2 such that for all n ≥ N 2 , P ( | P n − P | ( A N 1 ) < ϵ ) ≥ 1 − δ .
Revised: ϵ , and we can choose N 2 such that for all n ≥ N 2 , P ( | P n − P | ( A N 1 ) < ϵ ) ≥ 1 − δ .

Original: Then for all n ≥ max { N 1 , N 2 } , with probability 1 − δ ,
Revised: Then for all n ≥ max { N 1 , N 2 } , with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence P n ( A n ) → 0 in probability.
Revised: Hence P n ( A n ) → 0 in probability.

Original: Claim 10 .
Revised: Claim 10 .

Original: Suppose Assumption 1, 2, 3, A1 hold.
Revised: Suppose Assumption 1, 2, A1, A2 hold.

Original: [Equation]
Revised: (1) (cid:12)(cid:12) Q m (cid:0) ˆ p − 1 h n [ c X , ∞ ) ∩ ˆ q − 1 h m

Original:  in probability .[Equation] supp
Revised: [ c Y , ∞ ) (cid:1) − Q m (supp( P ) ∩ supp( Q )) (cid:12)(cid:12) → 0 in probability .

Original: [Equation]
Revised: (2) (cid:12)(cid:12) Q m (cid:0) ˆ q − 1 h m [ c Y , ∞ ) (cid:1) − Q

Original:  Q [Equation]
Revised: m (supp( Q ))

Original:  → 0 in probability .
Revised: (cid:12)(cid:12) → 0 in probability .

Original: (3) Q m (cid:0) ˆ p − 1 h n [ c X , ∞ ) ∩ ˆ q − 1 h m [ c Y , ∞ ) (cid:1) → Q (supp( P )) in probability .[Equation] ˆ [Equation]
Revised: (3) Q m (cid:0) ˆ p − 1 h n [ c X , ∞ ) ∩ ˆ q − 1 h m [ c Y , ∞ ) (cid:1) → Q (supp( P )) in probability .

Original: 1 h

Original: [Equation]

Original: Y

Original: [Equation] ∞ [Equation] → 1 in probability .
Revised: (4) Q m (cid:0) ˆ q − 1 h m [ c Y , ∞ ) (cid:1) → 1 in probability .

Original: Proof.
Revised: Proof.

Original: From Lemma 7, with high probability,
Revised: From Lemma 7, with high probability,

Original: [Equation]
Revised: [Equation]

Original: Then from the first inequality, combining with Claim 8 gives
Revised: Then from the first inequality, combining with Claim 8 gives

Original: [Equation]
Revised: [Equation]

Original: And from the second inequality, combining with Claim 8 gives
Revised: And from the second inequality, combining with Claim 8 gives

Original: [Equation]
Revised: [Equation]

Original: And hence
Revised: And hence

Original: [Equation]
Revised: [Equation]

Original: Now, note that from Lemma 5 implies that for all x ∈ supp( P ) , lim inf n p h n ( x ) > 0 , so p h n ( x ) > 2 c P for large enough n .
Revised: Now, note that from Lemma 5 implies that for all x ∈ supp( P ) , lim inf n p h n ( x ) > 0 , so p h n ( x ) > 2 c P for large enough n .

Original: And hence supp( P
Revised: And hence supp( P

Original: [Equation]
Revised: [Equation]

Original: \ p
Revised: \ p

Original: [Equation]
Revised: [Equation]

Original: 1 h n [2 c P , ∞ ) → ∅ .
Revised: 1 h n [2 c P , ∞ ) → ∅ .

Original: And similar argument holds for supp( Q ) \ q − 1 h m [2 c Q , ∞ ) , so supp( Q ) \ q − 1 h m [2 c Q , ∞ )
Revised: And similar argument holds for supp( Q ) \ q − 1 h m [2 c Q , ∞ ) , so supp( Q ) \ q − 1 h m [2 c Q , ∞ )

Original: → ∅ as well.
Revised: → ∅ as well.

Original: Then from Lemma 9,
Revised: Then from Lemma 9,

Original: [Equation]
Revised: [Equation]

Original: Also, since K has compact support, for any x / ∈ supp( P ) , x / ∈ supp( P h n ) once h n < d ( x, supp( P )) .
Revised: Also, since K has compact support, for any x / ∈ supp( P ) , x / ∈ supp( P h n ) once h n < d ( x, supp( P )) .

Original: Hence supp( P h n ) \ supp( P ) → ∅ , and similarly supp( Q h m ) \ supp( Q ) → ∅ as well.
Revised: Hence supp( P h n ) \ supp( P ) → ∅ , and similarly supp( Q h m ) \ supp( Q ) → ∅ as well.

Original: Then again with Lemma 9,
Revised: Then again with Lemma 9,

Original: [Equation]
Revised: [Equation]

Original: And hence
Revised: And hence

Original: [Equation]
Revised: [Equation]

Original: This can be done similarly to (1).
Revised: This can be done similarly to (1).

Original: Lemma 9 (1) gives that with probability 1 − δ ,
Revised: Lemma 9 (1) gives that with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence combining with (1) gives the desired result.
Revised: Hence combining with (1) gives the desired result.

Original: Lemma 9 (1) gives that with probability 1 − δ ,
Revised: Lemma 9 (1) gives that with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence combining with (2) gives the desired result.
Revised: Hence combining with (2) gives the desired result.

Original: Proof of Proposition 2.
Revised: Proof of Proposition 1.

Original: Now this is a combination of Claim 10 (1) (2).
Revised: Now this is a combination of Claim 10 (1) (2).

Original: Proof of Theorem 3.
Revised: Proof of Theorem 2.

Original: Now this is a combination of Claim 10 (3) (4).
Revised: Now this is a combination of Claim 10 (3) (4).

Original: E E XPERIMENTAL D ETAILS E. I MPLEMENTATION DETAILS OF EMBEDDING
Revised: E P HILOSOPHY OF OUR METRIC & P RACTICAL S CENARIOS E. P HILOSOPHY OF OUR METRIC

Revised: All evaluation metrics have different resolutions and properties. Here, we designed our proposed metric with the philosophy of evaluating the performance more conservatively based on (topologically and statistically) certain things. More specifically, in a real situation, there may be outliers in the data or samples we receive, noise may be present, and many other problems may arise due to various other unexpected causes.

Revised: In these situations, two approaches can be used in the assessment.

Revised: One is to accept ignorance and use all the data together, the other is to systematically select and exclude as much unreliable information as possible and only use reliable information.

Revised: We chose the latter because we thought seeing a conservatively consistent result had its own merits (At least we think our approach is worth investigating, showing different aspects that have not been explored before).

Revised: E. P RACTICAL S CENARIOS

Revised: From this perspective, we present two examples of realistic situations where outliers exist in the data and filtering out them can have a significant impact on proper model analysis and evaluation.

Revised: With real data, there are many cases where outliers are introduced into the data due to human error (Pleiss et al., 2020; Li et al., 2022).

Revised: Taking the simplest MNIST as an example, suppose our task of interest is to generate 4.

Revised: Since image number 7 is included in data set number 4 due to incorrect labeling (see

Revised: Figure 1 of (Pleiss et al., 2020)), the support of the real data in the feature space can be overestimated by such outliers, leading to an unfair evaluation of generative models (as in Section 5.1.2 and 5.2.3);

Revised: That is, the sample generated with weird noise may be in the overestimated support, and existing metrics without taking into account the reliability of the support could not penalize this, giving a good score to a poorly performing generator.

Revised: A similar but different example is when noise or distortions in the captured data (unfortunately) behave adversely on the feature embedding network used by the current evaluation metrics (as in

Revised: Section 5.1.2 and 5.2.3); e.g., visually it is the number 7, but it is mismapped near the feature space where there are usually 4 and becomes an outlier.

Revised: Then the same problem as above may occur.

Revised: Note that in these simple cases, where the definition of outliers is obvious with enough data, one could easily examine the data and exclude outliers a priori to train a generative model.

Revised: In the case of more complicated problems such as the medical field (Li et al., 2022), however, it is often not clear how outliers are to be defined.

Revised: Moreover, because data are often scarce, even outliers are very useful and valuable in practice for training models and extracting features, making it difficult to filter outliers in advance and decide not to use them.

Revised: On the other hand, we also provide an example where it is very important to filter out outliers in the generator sample and then evaluate them.

Revised: To evaluate the generator, samples are generated by sampling from the preset latent space (typically Gaussian).

Revised: Even after training is complete and the generators’ outputs are generally fine, there’s a latent area where generators aren’t fully trained.

Revised: Note that latent space sampling may contain samples from regions that the generator does not cover well during training ("unfortunate outlier").

Revised: When unfortunate outliers are included, the existing evaluation metrics may underestimate or overestimate the generator’s performance than its general performance.

Revised: (To get around this, it is necessary to try this evaluation several times to statistically stabilize it, but this requires a lot of computation and becomes impractical, especially when the latent space dimension is high.)

Revised: Especially considering the evaluation scenario in the middle of training, the above situation is likely to occur due to frequent evaluation, which can interrupt training or lead to wrong conclusions.

Revised: On the other hand, we can expect that our metric will be more robust against the above problem since it pays more attention to the core (samples that form topologically meaningful structures) generation performance of the model.

Revised: F E XPERIMENTAL D ETAILS F. I MPLEMENTATION DETAILS OF EMBEDDING

Original: We summarize the detailed information of our embedding networks implemented for the experiments.
Revised: We summarize the detailed information of our embedding networks implemented for the experiments.

Original: In Figure 2, 3, 4, 5, A1, and 6, P&R and D&C are computed from the features mapped by an ImageNet pre-trained VGG16, and TopP&R is computed from features placed in R 32 with additional random linear projection.
Revised: In Figure 2, 3, 4, 5, A2, and 6, P&R and D&C are computed from the features of ImageNet pre-trained VGG16 (fc2 layer), and TopP&R is computed from features placed in R 32 with additional random linear projection.

Original: In the experiment in Figure 5, the SwAV embedder is additionally considered.
Revised: In the experiment in Figure 5, the SwAV embedder is additionally considered.

Original: We implement ImageNet pre-trained InceptionV3, VGG16, and SwAV as embedding networks  to compare the ranking of GANs in Table 1.
Revised: We implement ImageNet pre-trained InceptionV3 (fc layer), VGG16 (fc2 layer), and SwAV as embedding networks with random linear projection to 32 dimensional feature space to compare the ranking of GANs in Table 1.

Original: The random projection is characterized by preserving the information about distances and homological features defined in the VGG16 feature space by Johnson LinenstraussLemma (Johnson et al., 1986). E. M EAN HAMMING DISTANCE
Revised: The random projection is characterized by preserving the information about distances and homological features defined in the higher dimensional spaces by Johnson Linenstrauss

Original: Hamming distance (HD) (Hamming, 1950) is a metric that counts the number of items with different ranks in A and B and measures how much proportion differs in the overall order, i.e. for A i ∈ A and B i ∈ B ,

Original: HD ( A, B ) : = (cid:80) ni =1 1 { k : A i ̸ = B i } /n where n is the number of items in list A or
Revised: Lemma (Johnson et al., 1986).

Original: B, and k is the number of differently ordered elements.

Original: The mean HD is calculated as follows to measure the average distances of three ordered lists: Given three ordered lists A, B, and C , ¯

Original: HD = ( HD ( A, B ) + HD ( A, C )

Original: + HD ( B, C )) /

Original: E.
Revised: F. C HOICE OF CONFIDENCE LEVEL

Original: C HOICE OF CONFIDENCE LEVEL

Original: For the confidence level α , we would like to point out that α is not the usual hyperparameter to be tuned: It has a statistical interpretation of the probability or the level of confidence to allow error, noise, etc.
Revised: For the confidence level α , we would like to point out that α is not the usual hyperparameter to be tuned: It has a statistical interpretation of the probability or the level of confidence to allow error, noise, etc.

Original: The most popular choices are α = 0 .
Revised: The most popular choices are α = 0 .

Original: 05 , 0 . 01 , leading to 90%, 95%, 99% confidence.
Revised: 05 , 0 . 01 , leading to 90%, 95%, 99% confidence.

Original: We used α = 0 .
Revised: We used α = 0 .

Original: 1 throughout our experiments.
Revised: 1 throughout our experiments.

Original: E. E STIMATION OF B ANDWIDTH PARAMETER
Revised: F. E STIMATION OF B ANDWIDTH PARAMETER

Original: As we discussed in section 2, since TopP&R estimates the manifold through KDE with kernel bandwidth parameter h , we need to approximate it.
Revised: As we discussed in section 2, since TopP&R estimates the manifold through KDE with kernel bandwidth parameter h , we need to approximate it.

Original: The estimation techniques for h are as follows: (1) a method of selecting h that maximizes the survival time ( S ( h ) ) or the number of significant homological features ( N ( h ) ) based on information obtained about persistent homology using the filtration method, (2) a method using the median of the k-nearest neighboring distances between features obtained by the balloon estimator (for more details, please refer to  Wagner et al.
Revised: The estimation techniques for h are as follows: ( a ) a method of selecting h that maximizes the survival time ( S ( h ) ) or the number of significant homological features ( N ( h ) ) based on information obtained about persistent homology using the filtration method, ( b ) a method using the median of the k-nearest neighboring distances between features obtained by the balloon estimator (for more details, please refer to Chazal et al. (2017), Wagner et al.

Original: (2012) and Terrell & Scott (1992)).
Revised: (2012), and Terrell & Scott (1992)).

Original: Note that, the bandwidth h for all the experiments in this paper are estimated via Balloon Bandwidth Estimator.
Revised: Note that, the bandwidth h for all the experiments in this paper are estimated via Balloon Bandwidth Estimator.

Original: 9: end
Revised: For (a) , following the notation in Section A, let the i th homological feature of persistent diagram be ( b i , d i ) , then we define its life length as l i ( h ) = d i − b i at kernel bandwidth h . With confidence band c α ( h ) , we select h that maximizes one of the following two quantities:

Revised: [Equation]

Original: E. E XPLICIT VALUES OF BANDWIDTH PARAMETER h 
Revised: Note that, we denote the confidence band c α as c α ( h ) considering the kernel bandwidth parameter h of KDE in Algorithm 1. For (b) , the balloon bandwidth estimator is defined as bellow: end F. M EAN HAMMING DISTANCE

Revised: Hamming distance (HD) (Hamming, 1950) counts the number of items with different ranks between

Original: Since our metric adaptively reacts to the given samples of P and Q
Revised: A and B, then measures how much proportion differs in the overall order, i.e. for A i ∈ A and B i ∈ B , HD ( A, B ) : = (cid:80) ni =1 1 { k : A i ̸

Original: [Table]
Revised: = B i } /n where n is the number of items in list A or B, and k is the number of differently ordered elements. The mean HD is calculated as follows to measure the average distances of three ordered lists: Given three ordered lists A, B, and C , ¯ HD = ( HD ( A, B ) + HD ( A, C ) + HD ( B, C )) /

Revised: F. E XPLICIT VALUES OF BANDWIDTH PARAMETER

Original:  we have two h s per experiment.For example, in the translation experiment (Figure 2) [Table]
Revised: Since our metric adaptively reacts to the given samples of P and Q , we have two h s per experiment.

Original:  there are 13 steps in total [Table] and each time we estimate h for P and Q , resulting in a total of 26 h s. To show them all at a glance, we have listed all values in one place.
Revised: For example, in the translation experiment (Figure 2), there are 13 steps in total, and each time we estimate h for P and Q , resulting in a total of 26 h s. To show them all at a glance, we have listed all values in one place.

Original: We will also provide the code that can reproduce the results in our experiments upon acceptance.
Revised: We will also provide the code that can reproduce the results in our experiments upon acceptance.

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: ψ is a parameter for the truncation trick and is first introduced in Brock et al. (2018) and Karras et al. We followed the approach in Brock et al.

Original: [Table]
Revised: (2018) and Karras et al.

Original: [Table]
Revised: GANs generate images using the noise input z , which follows the standard normal distribution N (0 , I ) or uniform distribution U ( − 1 , 1) . Suppose GAN inadvertently samples noise outside of distribution, then it is less likely to sample the image from the high density area of the image distribution p ( z ) defined in the latent space of GAN, which leads to generate an image with artifacts. The truncation trick takes this into account and uses the following truncated distribution. Let f be the mapping from the input to the latent space.

Original: [Table]
Revised: Let w = f ( z ) , and ¯ w = E

Original: [Table]
Revised: [ f ( z )] , where z is either from N (0 , I ) or U ( − 1 , 1) . Then we use w ′ = ¯ w + ψ ( w − ¯ w ) as a truncated latent vector. If the value of ψ increases, then the degree of truncation decreases which makes images have greater diversity but possibly lower fidelity.

Original: F A DDITIONAL E XPERIMENTS F. S URVIVABILITY OF MINORITY SETS IN THE LONG - TAILED DISTRIBUTION
Revised: G. S URVIVABILITY OF MINORITY SETS IN THE LONG - TAILED DISTRIBUTION

Original: [Table]
Revised: [Table]

Original: An important point to check in our proposed metric is the possibility that a small part of the total data, but containing important information, can be ignored by the confidence band.
Revised: An important point to check in our proposed metric is the possibility that a small part of the total data (i.e., minority sets), but containing important information, can be ignored by the confidence band.

Revised: We emphasize that since our metric takes topological features into account, even minority sets are not filtered conditioned that they have topologically significant structures.

Revised: We assume that signals or data that are minority sets have topological structures, but outliers exist far apart and lack a topological structure in general.

Original: To test this, we experimented with CIFAR10, which has 5,000 samples per class.
Revised: To test this [Table] we experimented with CIFAR10, which has 5,000 samples per class.

Original: We simulate a dataset with the majority set of six classes (2,000 samples per class, 12,000 total) and the minority set of four classes (500 samples per class, 2,000 total), and an ideal generator that exactly mimics the full data distribution.
Revised: We simulate a dataset with the majority set of six classes (2,000 samples per class, 12,000 total) and the minority set of four classes (500 samples per class, 2,000 total) [Table] and an ideal generator that exactly mimics the full data distribution.

Original: As shown in the Table 12, the samples in the minority set remained after the filtering process, meaning that the samples were sufficient to form a significant structure.
Revised: As shown in the Table A11, the samples in the minority set remained after the filtering process, meaning that the samples were sufficient to form a significant structure.

Original: BothD&C and
Revised: Both

Original:  TopP&R successfully evaluate the distribution for the ideal generator.
Revised: D&C and TopP&R successfully evaluate the distribution for the ideal generator.

Original: To check whether our metric reacts to the change in the distribution even with this harsh setting, we also carried out the mode decay experiment.
Revised: To check whether our metric reacts to the change in the distribution even with this harsh setting, we also carried out the mode decay experiment.

Original: We dropped the samples of the minority set from 500 to 100 per class, which can be interpreted as an 11.3% decrease in diversity relative to the full distribution (Given (1) ratio of the number of samples between majority and minority sets = 12 , 000 : 2 , 000 = 6 : 1 and (2) 80% decrease in samples per minority class, the true decay in the diversity is calculated as [Figure]
Revised: We dropped the samples of the minority set from 500 to 100 per class, which can be interpreted as an 11.3% decrease in diversity relative to the full distribution (Given (1) ratio of the number of samples between majority and minority sets = 12 , 000 : 2 , 000 = 6 :and (2) 80% decrease in samples per minority class, the true decay in the diversity is calculated as(1+6) × 0 . 8 = 11 .

Original:  with respect to the enitre samples).
Revised: 3% with respect to the enitre samples).

Original: Here, recall and coverage react somewhat less sensitively with their reduced diversities as 3 p.p. and 2 p.p., respectively, while TopR reacted most similarly (9 p.p.) to the ideal value. 
Revised: Here, recall and coverage react somewhat less sensitively with their reduced diversities as 3 p.p. and 2 p.p., respectively, while TopR reacted most similarly (9 p.p.) to the ideal value. In summary, TopP&R shows much more sensitiveness to the changes in data distribution like mode decay.

Revised: Thus, once the minority set has survived the filtering process, our metric is likely to be much more responsive than existing methods.

Original: F. SEQUENTIAL AND SIMULTANEOUS  
Revised: G. SEQUENTIAL AND SIMULTANEOUS MODE DROPPING WITH CIFAR-

Original: [Figure]
Revised: [Figure]

Original: [Figure]