Original: R OBUST M ANIFOLD E STIMATION A PPROACH FOR E VALUATING F IDELITY AND D IVERSITY
Revised: R OBUST M ANIFOLD E STIMATION A PPROACH FOR E VALUATING F IDELITY AND D IVERSITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a robust and reliable evaluation metric for generative models by introducing topological and statistical treatments for a rigorous support manifold estimation.
Revised: We propose a robust and reliable evaluation metric for generative models by introducing topological and statistical treatments for a rigorous support manifold estimation.

Original: Existing metrics, such as Inception Score (IS), Fréchet Inception
Revised: Existing metrics, such as Inception Score (IS), Fréchet Inception

Original: Distance (FID), and the variants of Precision and Recall ( P&R ), heavily rely on support manifolds that are estimated from sample features.
Revised: Distance (FID), and the variants of Precision and Recall ( P&R ), heavily rely on support manifolds that are estimated from sample features.

Original: However, the reliability of their estimation has not been seriously discussed (and overlooked) even though the quality of the evaluation entirely depends on it.
Revised: However, the reliability of their estimation has not been seriously discussed (and overlooked) even though the quality of the evaluation entirely depends on it.

Original: In this paper, we propose Topological Precision and Recall ( TopP&R , pronounced “topper”), which provides a systematic approach to estimating support manifolds, retaining only topologically and statistically important features with a certain level of confidence.
Revised: In this paper, we propose Topological Precision and Recall ( TopP&R , pronounced “topper”), which provides a systematic approach to estimating support manifolds, retaining only topologically and statistically important features with a certain level of confidence.

Original: This not only makes TopP&R strong for noisy features, but also provides statistical consistency.
Revised: This not only makes TopP&R strong for noisy features, but also provides statistical consistency.

Original: Our theoretical and experimental results show that TopP&R is robust to outliers and non-independent and identically distributed (Non-IID) perturbations, while accurately capturing the true trend of change in samples.
Revised: Our theoretical and experimental results show that TopP&R is robust to outliers and non-independent and identically distributed (Non-IID) perturbations, while accurately capturing the true trend of change in samples.

Original: To the best of our knowledge, this is the first evaluation metric focused on the robust estimation of the support manifold and provides its statistical consistency under noise.
Revised: To the best of our knowledge, this is the first evaluation metric focused on the robust estimation of the support manifold and provides its statistical consistency under noise.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In keeping with the remarkable improvements of deep generative models (Karras et al., 2019; 2020; 2021;
Revised: In keeping with the remarkable improvements of deep generative models (Karras et al., 2019; 2020; 2021;

Original: Brock et al., 2018; Ho et al., 2020; Kingma & Welling, 2013; Sauer et al., 2022; 2021; Kang &
Revised: Brock et al., 2018; Ho et al., 2020; Kingma & Welling, 2013; Sauer et al., 2022; 2021; Kang &

Original: Park, 2020), evaluation metrics that can well measure the performance of generative models have also been continuously developed (Salimans et al., 2016;
Revised: Park, 2020), evaluation metrics that can well measure the performance of generative models have also been continuously developed (Salimans et al., 2016;

Original: Heusel et al., 2017; Sajjadi et al., 2018;
Revised: Heusel et al., 2017; Sajjadi et al., 2018;

Original: Kynkäänniemi et al., 2019; Naeem et al., 2020).
Revised: Kynkäänniemi et al., 2019; Naeem et al., 2020).

Original: For instance, Inception Score (IS) (Salimans et al., 2016) measures the Kullback-Leibler divergence between the real and fake sample distributions in
Revised: For instance, Inception Score (IS) (Salimans et al., 2016) measures the Kullback-Leibler divergence between the real and fake sample distributions in

Original: VGG (Simonyan & Zisserman, 2014) feature space.
Revised: VGG (Simonyan & Zisserman, 2014) feature space.

Original: Fréchet Inception Score (FID)(Heusel et al., 2017) calculates the distance between the real and fake support manifolds using the estimated mean and variance under the multi-Gaussian assumption.
Revised: Fréchet Inception Score (FID)(Heusel et al., 2017) calculates the distance between the real and fake support manifolds using the estimated mean and variance under the multi-Gaussian assumption.

Original: The original Precision and Recall ( P&R ) (Sajjadi et al., 2018) and its variants (Kynkäänniemi et al., 2019; Naeem et al., 2020) were the first to separately investigate the fidelity and diversity aspects of generative performance.
Revised: The original Precision and Recall ( P&R ) (Sajjadi et al., 2018) and its variants (Kynkäänniemi et al., 2019; Naeem et al., 2020) were the first to separately investigate the fidelity and diversity aspects of generative performance.

Original: Considering the eminent progress of deep generative models based on these existing metrics, some may question why we need another evaluation study.
Revised: Considering the eminent progress of deep generative models based on these existing metrics, some may question why we need another evaluation study.

Original: In this paper, we argue that we need more reliable evaluation metrics now precisely, because deep generative models have reached sufficient maturity.
Revised: In this paper, we argue that we need more reliable evaluation metrics now precisely, because deep generative models have reached sufficient maturity.

Original: To provide a more accurate and comprehensive ideas and to illuminate a new direction of improvements in the generative field, we need a more robust and reliable evaluation metric.
Revised: To provide a more accurate and comprehensive ideas and to illuminate a new direction of improvements in the generative field, we need a more robust and reliable evaluation metric.

Original: In fact, it has been recently reported that even the most widely used evaluation metric, FID, sometimes doesn’t match with the expected perceptual quality, fidelity, and diversity, which means the metrics are not always working properly (Kynkäänniemi et al., 2022).
Revised: In fact, it has been recently reported that even the most widely used evaluation metric, FID, sometimes doesn’t match with the expected perceptual quality, fidelity, and diversity, which means the metrics are not always working properly (Kynkäänniemi et al., 2022).

Original: In addition to this, in practice, not only do generated samples but also real data in the wild often contain lots of artifacts, and these have been shown to seriously perturb the existing evaluation metrics, giving a false sense of improvements (Naeem et al., 2020; Kynkäänniemi et al., 2022).
Revised: In addition to this, in practice, not only do generated samples but also real data in the wild often contain lots of artifacts, and these have been shown to seriously perturb the existing evaluation metrics, giving a false sense of improvements (Naeem et al., 2020; Kynkäänniemi et al., 2022).

Original: An ideal evaluation metric must capture the real signal of the data, while being robust to noise.
Revised: An ideal evaluation metric must capture the real signal of the data, while being robust to noise.

Original: Note that there is an inherent tension in developing metrics that meets these goals.
Revised: Note that there is an inherent tension in developing metrics that meets these goals.

Original: On one hand, the metric should be sensitive enough so that it can capture real signals lurking in data.
Revised: On one hand, the metric should be sensitive enough so that it can capture real signals lurking in data.

Original: On the other hand, it must ignore noises that hide the signal.
Revised: On the other hand, it must ignore noises that hide the signal.

Original: However, sensitive metrics are inevitably susceptible to noise
Revised: However, sensitive metrics are inevitably susceptible to noise

Original: [Figure]
Revised: [Figure]

Original: (𝑃
Revised: (𝑃

Original: [Figure]
Revised: [Figure]

Original: 𝑠𝑢𝑝𝑝(𝑄)
Revised: 𝑠𝑢𝑝𝑝(𝑄)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: ̂𝑝 !
Revised: ̂𝑝 !

Original: (#) ̂𝑝 !
Revised: (#) ̂𝑝 !

Original: (%) ̂𝑝 !
Revised: (%) ̂𝑝 !

Original: (&) 𝜃
Revised: (&) 𝜃

Original: [Figure]
Revised: [Figure]

Original: ̂𝑝 !
Revised: ̂𝑝 !

Original: − ̂𝑝 !
Revised: − ̂𝑝 !

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ̂𝑝 !
Revised: ̂𝑝 !

Original: − ̂𝑝 !
Revised: − ̂𝑝 !

Original: [Figure]
Revised: [Figure]

Original: to some extent.
Revised: to some extent.

Original: To address this, one needs a systematic way to answer the following two questions: 1) what is signal and what is noise? and 2) how do we draw a line between them?
Revised: To address this, one needs a systematic way to answer the following two questions: 1) what is signal and what is noise? and 2) how do we draw a line between them?

Original: One solution can be to use the idea of statistical inference and topological data analysis (TDA).
Revised: One solution can be to use the idea of statistical inference and topological data analysis (TDA).

Original: Topological data analysis (TDA) (Carlsson, 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data.
Revised: Topological data analysis (TDA) (Carlsson, 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data.

Original: A key object in
Revised: A key object in

Original: TDA is persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.
Revised: TDA is persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.

Original: It observes how long each homological feature would survive across different resolutions through the lens of topology and provides a way to quantify its importance; That is, if some features persist longer than others, we consider the features and the data that compose them as a topological signal and vice versa as noise.
Revised: It observes how long each homological feature would survive across different resolutions through the lens of topology and provides a way to quantify its importance; That is, if some features persist longer than others, we consider the features and the data that compose them as a topological signal and vice versa as noise.

Original: Then, statistical inference provides a systematic way to establish statistical thresholds that separate the topological signal from topological noise with statistical interpretation.
Revised: Then, statistical inference provides a systematic way to establish statistical thresholds that separate the topological signal from topological noise with statistical interpretation.

Original: In this paper, we propose to combine these ideas to form a more robust and compact feature manifold and overcome various issues from the conventional metrics.
Revised: In this paper, we propose to combine these ideas to form a more robust and compact feature manifold and overcome various issues from the conventional metrics.

Original: Our main contributions are as follows: we introduce (1) an approach to directly estimate a support manifold via Kernel Density Estimator (KDE) derived under topological conditions; (2) a new metric that is robust to outliers while reliably detecting the change of distributions on various scenarios; and (3) a theoretical guarantee of consistency with robustness under very weak assumptions that is suitable for high dimensional data.
Revised: Our main contributions are as follows: we introduce (1) an approach to directly estimate a support manifold via Kernel Density Estimator (KDE) derived under topological conditions; (2) a new metric that is robust to outliers while reliably detecting the change of distributions on various scenarios; and (3) a theoretical guarantee of consistency with robustness under very weak assumptions that is suitable for high dimensional data.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: To lay the foundation for our theoretical analysis, we introduce the main idea of persistent homology and its confidence estimation techniques that bring the benefit of using topological and statistical tools for addressing uncertainty in samples.
Revised: To lay the foundation for our theoretical analysis, we introduce the main idea of persistent homology and its confidence estimation techniques that bring the benefit of using topological and statistical tools for addressing uncertainty in samples.

Original: In later sections, we use these tools to analyze the effects of outliers in evaluating generative models and provide more rigorous way of scoring the samples based on the confidence level we set.
Revised: In later sections, we use these tools to analyze the effects of outliers in evaluating generative models and provide more rigorous way of scoring the samples based on the confidence level we set.

Original: For space reasons, we only provide a brief overview of the concepts that are relevant to this work and refer the reader to Section A or (Edelsbrunner & Harer, 2010; Chazal & Michel, 2021; Wasserman, 2018; Hatcher, 2002) for further details.
Revised: For space reasons, we only provide a brief overview of the concepts that are relevant to this work and refer the reader to Section A or (Edelsbrunner & Harer, 2010; Chazal & Michel, 2021; Wasserman, 2018; Hatcher, 2002) for further details.

Original: 2. N OTATION
Revised: 2. N OTATION

Original: For any x and r > 0 , we use the notation B d ( x, r )
Revised: For any x and r > 0 , we use the notation B d ( x, r )

Original: = { y : d ( y, x ) < r } be the open ball in distance d of radius r .
Revised: = { y : d ( y, x ) < r } be the open ball in distance d of radius r .

Original: We also write B ( x, r ) when d is understood from context.
Revised: We also write B ( x, r ) when d is understood from context.

Original: For a distribution P on R d , we let supp( P ) :
Revised: For a distribution P on R d , we let supp( P ) :

Original: = { x ∈ R d : P ( B ( x, r )) > 0 for all r > 0 } be the support of P .
Revised: = { x ∈ R d : P ( B ( x, r )) > 0 for all r > 0 } be the support of P .

Original: Throughout the paper, we refer to supp(P) as support manifold of P , or simply support, or manifold, but we don’t necessarily require the (geometrical) manifold structure on supp(P) .
Revised: Throughout the paper, we refer to supp(P) as support manifold of P , or simply support, or manifold, but we don’t necessarily require the (geometrical) manifold structure on supp(P) .

Original: For a kernel function
Revised: For a kernel function

Original: K : R d → R , a dataset X =
Revised: K : R d → R , a dataset X =

Original: { X 1 , . . . ,
Revised: { X 1 , . . . ,

Original: X n } ⊂ R d and bandwidth h > 0 , we let the kernel density estimator (KDE) as ˆ p h ( x ) :
Revised: X n } ⊂ R d and bandwidth h > 0 , we let the kernel density estimator (KDE) as ˆ p h ( x ) :

Original: = 1 nh d (cid:80) ni =1 K (cid:0) x − X i h (cid:1) , and we let the average KDE as p h :=
Revised: = 1 nh d (cid:80) ni =1 K (cid:0) x − X i h (cid:1) , and we let the average KDE as p h :=

Original: E [ˆ p h ] .
Revised: E [ˆ p h ] .

Original: We denote by P , Q the probability distributions in R d of real data and generated samples, respectively.
Revised: We denote by P , Q the probability distributions in R d of real data and generated samples, respectively.

Original: And we use X = { X 1 , . . .
Revised: And we use X = { X 1 , . . .

Original: X n } ⊂ R d and Y =
Revised: X n } ⊂ R d and Y =

Original: { Y 1 , . . .
Revised: { Y 1 , . . .

Original: , Y m } ⊂ R d for real data and generated samples possibly with noise, respectively.
Revised: , Y m } ⊂ R d for real data and generated samples possibly with noise, respectively.

Original: 2. C ONFIDENCE BAND ESTIMATION
Revised: 2. C ONFIDENCE BAND ESTIMATION

Original: Statistical inference has recently been developed for topological data analysis (Chazal et al., 2013; 2015; Fasy et al., 2014).
Revised: Statistical inference has recently been developed for topological data analysis (Chazal et al., 2013; 2015; Fasy et al., 2014).

Original: Topological data analysis consists of features reflecting topological characteristics of data, and it is of question to distinguish features that are indeed from geometrical structures and features that are insignificant or due to noise.
Revised: Topological data analysis consists of features reflecting topological characteristics of data, and it is of question to distinguish features that are indeed from geometrical structures and features that are insignificant or due to noise.

Original: To statistically separate topologically significant features from topological noise, we use a confidence band.
Revised: To statistically separate topologically significant features from topological noise, we use a confidence band.

Original: Given the significance level α , let confidence band
Revised: Given the significance level α , let confidence band

Original: c
Revised: c

Original: X be the bootstrap bandwidth of ∥ ˆ p h − p ∗ h ∥ ∞ .
Revised: X be the bootstrap bandwidth of ∥ ˆ p h − p ∗ h ∥ ∞ .

Original: Then it satisfies lim inf n →∞ P ( ∥ ˆ p
Revised: Then it satisfies lim inf n →∞ P ( ∥ ˆ p

Original: h − p h
Revised: h − p h

Original: ∥ ∞ < c X ) ≥ 1 − α , as in Proposition 4 in Section C.
Revised: ∥ ∞ < c X ) ≥ 1 − α , as in Proposition 4 in Section C.

Original: This confidence band can be used to determine simultaneously significant topological features while filtering out noise features.
Revised: This confidence band can be used to determine simultaneously significant topological features while filtering out noise features.

Original: The algorithm for computing c X is described below.
Revised: The algorithm for computing c X is described below.

Original: θ with √ n || ˆ p h − ˆ p ∗ h || ∞ 13: end for 14: # grid search for the confidence band 15: for q ∈ [ min (ˆ θ ) , max (ˆ θ )] do 16: count =17: for element ∈
Revised: θ with √ n || ˆ p h − ˆ p ∗ h || ∞ 13: end for 14: # grid search for the confidence band 15: for q ∈ [ min (ˆ θ ) , max (ˆ θ )] do 16: count =17: for element ∈

Original: ˆ θ do 18: # count significant difference 19: if element > q then 20: count = count +21: end if 22: end for 23:
Revised: ˆ θ do 18: # count significant difference 19: if element > q then 20: count = count +21: end if 22: end for 23:

Original: # define the band threshold 24: if count /k
Revised: # define the band threshold 24: if count /k

Original: ≈
Revised: ≈

Original: α
Revised: α

Original: then 25: q α = q 26: end if 27: end for 28: # define estimated confidence band 29: c α = q α / √ n
Revised: then 25: q α = q 26: end if 27: end for 28: # define estimated confidence band 29: c α = q α / √ n

Original:  R OBUST SUPPORT MANIFOLD ESTIMATION FOR RELIABLE EVALUATION
Revised:  R OBUST SUPPORT MANIFOLD ESTIMATION FOR RELIABLE EVALUATION

Original: Current evaluation metrics for generative models typically rely on strong regularity conditions.
Revised: Current evaluation metrics for generative models typically rely on strong regularity conditions.

Original: For example, they assume samples are well-curated without outliers or adversarial perturbation, real or generative models have bounded densities, etc.
Revised: For example, they assume samples are well-curated without outliers or adversarial perturbation, real or generative models have bounded densities, etc.

Original: However, practical scenarios are wild: both real and generated samples can be corrupted with noise from various sources, and the real data can be very sparsely distributed without density.
Revised: However, practical scenarios are wild: both real and generated samples can be corrupted with noise from various sources, and the real data can be very sparsely distributed without density.

Original: In this work, we consider more general and practical situations, wherein both real and generated samples can have noises that come from sampling procedure, remained uncertainty due to data or model, etc.
Revised: In this work, we consider more general and practical situations, wherein both real and generated samples can have noises that come from sampling procedure, remained uncertainty due to data or model, etc.

Original: 3. T OPOLOGICAL PRECISION AND RECALL
Revised: 3. T OPOLOGICAL PRECISION AND RECALL

Original: In the ideal case where we have full access to the probability distributions P and Q , we define the precision and the recall of distributions as precision P ( Q ) :
Revised: In the ideal case where we have full access to the probability distributions P and Q , we define the precision and the recall of distributions as precision P ( Q ) :

Original: = Q (supp( P )) , recall Q ( P ) :
Revised: = Q (supp( P )) , recall Q ( P ) :

Original: = P (supp( Q )) .
Revised: = P (supp( Q )) .

Original: These correspond to the max precision and the max recall in Sajjadi et al.
Revised: These correspond to the max precision and the max recall in Sajjadi et al.

Original: We tweak the precision as precision P ( Y ) =
Revised: We tweak the precision as precision P ( Y ) =

Original: Q (supp( P ) ∩ supp( Q )) /Q
Revised: Q (supp( P ) ∩ supp( Q )) /Q

Original: (supp( Q )) , and define the precision of data points as precision P ( Y
Revised: (supp( Q )) , and define the precision of data points as precision P ( Y

Original: [Equation]
Revised: [Equation]

Original: :=
Revised: :=

Original: [Equation]
Revised: [Equation]

Original: which is just replacing the distribution Q by the empirical distribution 1 m (cid:80) mj
Revised: which is just replacing the distribution Q by the empirical distribution 1 m (cid:80) mj

Original: =1 δ Y j of Y in the precision.
Revised: =1 δ Y j of Y in the precision.

Original: We similarly define the recall of data points as
Revised: We similarly define the recall of data points as

Original: [Equation]
Revised: [Equation]

Original: However, in practice, supp( P ) and supp( Q ) are not known a priori and need to be estimated, and since we allow noise, these estimates should be robust to noise.
Revised: However, in practice, supp( P ) and supp( Q ) are not known a priori and need to be estimated, and since we allow noise, these estimates should be robust to noise.

Original: For this, we use the kernel density estimator (KDE) and the bootstrap bandwidth to robustly estimate the support.
Revised: For this, we use the kernel density estimator (KDE) and the bootstrap bandwidth to robustly estimate the support.

Original: Given h n > 0 and a significance level α ∈ (0 , 1) , we use the KDE ˆ p h n ( x ) :
Revised: Given h n > 0 and a significance level α ∈ (0 , 1) , we use the KDE ˆ p h n ( x ) :

Original: = 1 nh dn (cid:80) ni =1 K (cid:16) x − X i h n (cid:17) of X , and we use the bootstrap bandwidth c X of (cid:13)(cid:13)
Revised: = 1 nh dn (cid:80) ni =1 K (cid:16) x − X i h n (cid:17) of X , and we use the bootstrap bandwidth c X of (cid:13)(cid:13)

Original: ˆ p h n − ˆ p ∗ h n (cid:13)(cid:13) ∞ from Section 2.
Revised: ˆ p h n − ˆ p ∗ h n (cid:13)(cid:13) ∞ from Section 2.

Original: Then we estimate the support of P by the superlevel set at c X as ˆ supp
Revised: Then we estimate the support of P by the superlevel set at c X as ˆ supp

Original: [Equation]
Revised: [Equation]

Original: P ) =
Revised: P ) =

Original: [Equation]
Revised: [Equation]

Original: h n [ c X , ∞ ) .
Revised: h n [ c X , ∞ ) .

Original: Similarly, we let ˆ q h m ( x ) :=
Revised: Similarly, we let ˆ q h m ( x ) :=

Original: [Equation]
Revised: [Equation]

Original: be the KDE of Y and let c Y be the bootstrap bandwidth of (cid:13)(cid:13)
Revised: be the KDE of Y and let c Y be the bootstrap bandwidth of (cid:13)(cid:13)

Original: ˆ q h m − ˆ q ∗ h m (cid:13)(cid:13) ∞ , and then we use ˆ supp( Q )
Revised: ˆ q h m − ˆ q ∗ h m (cid:13)(cid:13) ∞ , and then we use ˆ supp( Q )

Original: =
Revised: =

Original: ˆ q − 1 h m [ c Y , ∞ ) .
Revised: ˆ q − 1 h m [ c Y , ∞ ) .

Original: Using the superlevel set at c X allows to filter out noise whose KDE values are likely to be small.
Revised: Using the superlevel set at c X allows to filter out noise whose KDE values are likely to be small.

Original: For the robust estimates of the precision, we apply the support estimates to the precision of data points, and define the topological precision ( TopP ) as
Revised: For the robust estimates of the precision, we apply the support estimates to the precision of data points, and define the topological precision ( TopP ) as

Original: [Equation]
Revised: [Equation]

Original: And we similarly define the topological recall ( TopR ) as
Revised: And we similarly define the topological recall ( TopR ) as

Original: [Equation]
Revised: [Equation]

Original: 3. B ANDWIDTH ESTIMATION USING BOOTSTRAPPING
Revised: 3. B ANDWIDTH ESTIMATION USING BOOTSTRAPPING

Original: Using the bootstrap bandwidth c X as threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) .
Revised: Using the bootstrap bandwidth c X as threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) .

Original: As we have seen in Section 2, the bootstrap bandwidth c X acts as a threshold for filtering out the topological noise in topological data analysis.
Revised: As we have seen in Section 2, the bootstrap bandwidth c X acts as a threshold for filtering out the topological noise in topological data analysis.

Original: Analogously, using c X as a threshold allows to robustly estimating supp( P ) .
Revised: Analogously, using c X as a threshold allows to robustly estimating supp( P ) .

Original: In particular, when X i is an outlier, its KDE value ˆ p h ( X i ) is likely to be small, and the KDE values at the connected component generated by X i is likely to be small as well.
Revised: In particular, when X i is an outlier, its KDE value ˆ p h ( X i ) is likely to be small, and the KDE values at the connected component generated by X i is likely to be small as well.

Original: So those components from outliers are likely to be removed in the estimated support ˆ p − 1 h [ c X , ∞ ) .
Revised: So those components from outliers are likely to be removed in the estimated support ˆ p − 1 h [ c X , ∞ ) .

Original: Hence, the estimated support denoises topological noise from outliers and robustly estimates supp( P ) .
Revised: Hence, the estimated support denoises topological noise from outliers and robustly estimates supp( P ) .

Original: See Section B for more detailed explanation.
Revised: See Section B for more detailed explanation.

Original: Now that we are only left with topological features of high confidence, this allows us to draw analogies to confidence intervals in statistical analysis, where the uncertainty of the samples is treated by setting the level of confidence.
Revised: Now that we are only left with topological features of high confidence, this allows us to draw analogies to confidence intervals in statistical analysis, where the uncertainty of the samples is treated by setting the level of confidence.

Original: In the next section, we show that TopP&R not only gives a more reliable evaluation score for generated samples but also has a good theoretical properties.
Revised: In the next section, we show that TopP&R not only gives a more reliable evaluation score for generated samples but also has a good theoretical properties.

Original:  C ONSISTENCY WITH ROBUSTNESS OF T OP P&R
Revised:  C ONSISTENCY WITH ROBUSTNESS OF T OP P&R

Original: The key properties of TopP&R is consistency with robustness.
Revised: The key properties of TopP&R is consistency with robustness.

Original: The consistency ensures that, the precision and the recall we compute from the data approaches the precision and the recall from the distribution as we have more samples.
Revised: The consistency ensures that, the precision and the recall we compute from the data approaches the precision and the recall from the distribution as we have more samples.

Original: The consistency allows to investigate the precision and the recall of the full distributions only with access to finite sampled data.
Revised: The consistency allows to investigate the precision and the recall of the full distributions only with access to finite sampled data.

Original: TopP&R achieves consistency with robustness, that is, the consistency holds with the data possibly corrupted by noise.
Revised: TopP&R achieves consistency with robustness, that is, the consistency holds with the data possibly corrupted by noise.

Original: This is due to the robust estimation of the support with the kernel density estimator with confidence bands.
Revised: This is due to the robust estimation of the support with the kernel density estimator with confidence bands.

Original: This section is devoted to the theoretical analysis of consistency of TopP&R with robustness.
Revised: This section is devoted to the theoretical analysis of consistency of TopP&R with robustness.

Original: We demonstrate the statistical model for the data and the noise.
Revised: We demonstrate the statistical model for the data and the noise.

Original: Let P , Q , X , Y be as in Notation in
Revised: Let P , Q , X , Y be as in Notation in

Original: Section 2, and let X 0 , Y 0 be real data and generated data without noise.
Revised: Section 2, and let X 0 , Y 0 be real data and generated data without noise.

Original: X , Y , X 0 , Y 0 are understood as multisets, i.e. , elements can be repeated.
Revised: X , Y , X 0 , Y 0 are understood as multisets, i.e. , elements can be repeated.

Original: We first assume that the uncorrupted data are IID.
Revised: We first assume that the uncorrupted data are IID.

Original: Assumption 1.
Revised: Assumption 1.

Original: The data X 0 = { X 01 , . . .
Revised: The data X 0 = { X 01 , . . .

Original: , X 0 n } and Y 

Revised: , X 0 n } and Y

Original: = { Y 01 , . . .
Revised: = { Y 01 , . . .

Original: , Y 0 m } are IID from P and Q , respectively.
Revised: , Y 0 m } are IID from P and Q , respectively.

Original: In practice, the data is often corrupted with noise.
Revised: In practice, the data is often corrupted with noise.

Original: We consider the adversarial noise, where some fraction of data are replaced with arbitrary point cloud data.
Revised: We consider the adversarial noise, where some fraction of data are replaced with arbitrary point cloud data.

Original: Assumption 2.
Revised: Assumption 2.

Original: Let { ρ k } k ∈
Revised: Let { ρ k } k ∈

Original: N
Revised: N

Original: be a sequence of nonnegative real numbers.
Revised: be a sequence of nonnegative real numbers.

Original: Then the observed data X and Y satisfies (cid:12)(cid:12) X\X 0 (cid:12)(cid:12) = nρ n and (cid:12)(cid:12) Y\Y 0 (cid:12)(cid:12) =
Revised: Then the observed data X and Y satisfies (cid:12)(cid:12) X\X 0 (cid:12)(cid:12) = nρ n and (cid:12)(cid:12) Y\Y 0 (cid:12)(cid:12) =

Original: mρ m .
Revised: mρ m .

Original: In the adversarial model, we control the level of noise by the fraction ρ , but do not assume other conditions such as IID or boundedness, to make our noise model very general and challenging.
Revised: In the adversarial model, we control the level of noise by the fraction ρ , but do not assume other conditions such as IID or boundedness, to make our noise model very general and challenging.

Original: For distributions, we assume that the order of probability volume decay P ( B ( x, r )) is at least r d .
Revised: For distributions, we assume that the order of probability volume decay P ( B ( x, r )) is at least r d .

Original: Assumption 3.
Revised: Assumption 3.

Original: For all x ∈ supp( P ) and y ∈ supp( Q ) ,
Revised: For all x ∈ supp( P ) and y ∈ supp( Q ) ,

Original: [Equation]
Revised: [Equation]

Original: Remark 1 .
Revised: Remark 1 .

Original: Assumption 3 is analogous to Assumption 2 of (Kim et al., 2019), but is weaker since the condition is pointwise on each x ∈ R d .
Revised: Assumption 3 is analogous to Assumption 2 of (Kim et al., 2019), but is weaker since the condition is pointwise on each x ∈ R d .

Original: And this condition is much weaker than assuming a density on R d : for example, a distribution supported on a low-dimensional manifold satisfies Assumption 3.
Revised: And this condition is much weaker than assuming a density on R d : for example, a distribution supported on a low-dimensional manifold satisfies Assumption 3.

Original: This provides a framework suitable for high dimensional data, since many times high dimensional data lies on a low dimensional structure hence its density on R d cannot exist.
Revised: This provides a framework suitable for high dimensional data, since many times high dimensional data lies on a low dimensional structure hence its density on R d cannot exist.

Original: See (Kim et al., 2019) for more detailed discussion.
Revised: See (Kim et al., 2019) for more detailed discussion.

Original: For kernel functions, we also assume weak condition, detailed in Assumption A1 in Section C. Under the data and the noise models, TopP&R achieves consistency with robustness.
Revised: For kernel functions, we also assume weak condition, detailed in Assumption A1 in Section C. Under the data and the noise models, TopP&R achieves consistency with robustness.

Original: That is, the estimated precision and recall is asymptotically correct with high probability even if up to a portion of 1 / √ n or 1 / √ m are replaced by adversarial noise.
Revised: That is, the estimated precision and recall is asymptotically correct with high probability even if up to a portion of 1 / √ n or 1 / √ m are replaced by adversarial noise.

Original: This is due to the robust estimation of the support with the kernel density estimator with the confidence band of the persistent homology.
Revised: This is due to the robust estimation of the support with the kernel density estimator with the confidence band of the persistent homology.

Original: Proposition 2.
Revised: Proposition 2.

Original: Suppose Assumption 1,2,3,A1 hold.
Revised: Suppose Assumption 1,2,3,A1 hold.

Original: Suppose h n → 0 , nh n → ∞ , nh − dn ρ 2 n → 0 , and similar relations hold for h m , ρ m .
Revised: Suppose h n → 0 , nh n → ∞ , nh − dn ρ 2 n → 0 , and similar relations hold for h m , ρ m .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: Under the same condition as in Proposition 2, |
Revised: Under the same condition as in Proposition 2, |

Original: [Equation]
Revised: [Equation]

Original: Our theoretical results in Proposition 2 and Theorem 3 are novel and important in several perspetives.
Revised: Our theoretical results in Proposition 2 and Theorem 3 are novel and important in several perspetives.

Original: These results are among the first theoretical guarantees for evaluation metrics for generative models as far as we are aware of.
Revised: These results are among the first theoretical guarantees for evaluation metrics for generative models as far as we are aware of.

Original: Also, as in Remark 1, assumptions are very weak and suitable for high dimensional data.
Revised: Also, as in Remark 1, assumptions are very weak and suitable for high dimensional data.

Original: Also, robustness to adversarial noise is provably guaranteed.
Revised: Also, robustness to adversarial noise is provably guaranteed.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: A good evaluation metric must correctly capture the changes of the underlying data distribution.
Revised: A good evaluation metric must correctly capture the changes of the underlying data distribution.

Original: To examine the performance of evaluation metrics, we carefully select a set of experiments for sanity checks.
Revised: To examine the performance of evaluation metrics, we carefully select a set of experiments for sanity checks.

Original: With toy and real image data, we check 1) how well the metric captures the true trend of underlying data distributions and 2) how well the metric resist perturbations applied to samples.
Revised: With toy and real image data, we check 1) how well the metric captures the true trend of underlying data distributions and 2) how well the metric resist perturbations applied to samples.

Original: 5. S ANITY CHECKS WITH TOY DATA
Revised: 5. S ANITY CHECKS WITH TOY DATA

Original: Following Naeem et al.
Revised: Following Naeem et al.

Original: (2020), we first examine how well the metric reflects the trend of Y moving away from X and whether it is suitable for finding mode-drop phenomena.
Revised: (2020), we first examine how well the metric reflects the trend of Y moving away from X and whether it is suitable for finding mode-drop phenomena.

Original: In addition to these, we newly design several experiments that can highlight TopP&R ’s favorable theoretical properties, such as consistency with robustness, in various scenarios.
Revised: In addition to these, we newly design several experiments that can highlight TopP&R ’s favorable theoretical properties, such as consistency with robustness, in various scenarios.

Original: 5.1. S HIFTING THE GENERATED FEATURE MANIFOLD
Revised: 5.1. S HIFTING THE GENERATED FEATURE MANIFOLD

Original: For this experiment, we generate samples for X ∼ N (0 , I ) and Y ∼ N ( µ 1 , I ) in R 64 where 1 is a vector of ones and I is an identity matrix.
Revised: For this experiment, we generate samples for X ∼ N (0 , I ) and Y ∼ N ( µ 1 , I ) in R 64 where 1 is a vector of ones and I is an identity matrix.

Original: We then examine how each metric responds to shifting
Revised: We then examine how each metric responds to shifting

Original: Y with µ ∈ [ − 1 , 1] while there are outliers at 3 ∈ R 64 for both X and Y (Figure 2).
Revised: Y with µ ∈ [ − 1 , 1] while there are outliers at 3 ∈ R 64 for both X and Y (Figure 2).

Original: Here, we find that both improved P&R and D&C behave pathologically when there are outliers.
Revised: Here, we find that both improved P&R and D&C behave pathologically when there are outliers.

Original: Since these
Revised: Since these

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  methods are based on the k-nearest neighbor algorithm and ignore the fact that there can be outliers in both real and fake data, they inevitably overestimate the underlying support when there are outliers.
Revised: ten trials. methods are based on the k-nearest neighbor algorithm and ignore the fact that there can be outliers in both real and fake data, they inevitably overestimate the underlying support when there are outliers.

Original: For example, when X lies between Y and the outlier at y = 3 , Recall returns a high-diversity score, even though the true supports of X and Y are actually far apart.
Revised: For example, when X lies between Y and the outlier at y = 3 , Recall returns a high-diversity score, even though the true supports of X and Y are actually far apart.

Original: In addition, P&R does not reach 1 in high dimensions even when X = Y .
Revised: In addition, P&R does not reach 1 in high dimensions even when X = Y .

Original: Naeem et al.
Revised: Naeem et al.

Original: (2020) circumvented these problems by proposing D&C that always use X (the real data distribution) as a reference point, which in most cases is assumed to have fewer outliers than Y (the fake data distribution).
Revised: (2020) circumvented these problems by proposing D&C that always use X (the real data distribution) as a reference point, which in most cases is assumed to have fewer outliers than Y (the fake data distribution).

Original: However, there is no guarantee that this will be the case in practice.
Revised: However, there is no guarantee that this will be the case in practice.

Original: When there is an outlier in X , D&C also returns an incorrect high-fidelity score at µ > 0 .
Revised: When there is an outlier in X , D&C also returns an incorrect high-fidelity score at µ > 0 .

Original: On the other hand, TopP&R shows a stable trend unaffected by outliers, demonstrating the robustness of our method.
Revised: On the other hand, TopP&R shows a stable trend unaffected by outliers, demonstrating the robustness of our method.

Original: 5.1. S EQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES
Revised: 5.1. S EQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES

Original: For this experiment, we consider the mixture of Gaussians with seven modes in R 64 .
Revised: For this experiment, we consider the mixture of Gaussians with seven modes in R 64 .

Original: We simulate mode-drop phenomena by gradually dropping all but one mode from the fake distribution Y that is initially identical to X (Figure 3).
Revised: We simulate mode-drop phenomena by gradually dropping all but one mode from the fake distribution Y that is initially identical to X (Figure 3).

Original: As in the illustration of mode-drop experiment, when the number of samples in a particular mode decreases, we kept the number of samples in X constant so that the same amount of decreased samples are supplemented to the first mode which leads fidelity to be fixed to 1.
Revised: As in the illustration of mode-drop experiment, when the number of samples in a particular mode decreases, we kept the number of samples in X constant so that the same amount of decreased samples are supplemented to the first mode which leads fidelity to be fixed to 1.

Original: From the result, we observe that the values of Precision fail to saturate, i.e. , mainly smaller than 1, and the Density fluctuates to a value greater than 1 indicating their instability and unboundedness.
Revised: From the result, we observe that the values of Precision fail to saturate, i.e. , mainly smaller than 1, and the Density fluctuates to a value greater than 1 indicating their instability and unboundedness.

Original: In terms of diversity, Recall does not respond to the simultaneous mode drop, nor does the improved metric Coverage show a fast decay as the reference line.
Revised: In terms of diversity, Recall does not respond to the simultaneous mode drop, nor does the improved metric Coverage show a fast decay as the reference line.

Original: Compared to these methods, TopP performs well, being held at the upperbound of 1 in sequential mode dropping, and
Revised: Compared to these methods, TopP performs well, being held at the upperbound of 1 in sequential mode dropping, and

Original: TopR also decreases closest to the reference line in simultaneous mode drops.
Revised: TopR also decreases closest to the reference line in simultaneous mode drops.

Original: 5.1. T OLERANCE TO N ON -IID PERTURBATIONS
Revised: 5.1. T OLERANCE TO N ON -IID PERTURBATIONS

Original: Robustness to perturbations is another important aspect we should consider when designing a metric.
Revised: Robustness to perturbations is another important aspect we should consider when designing a metric.

Original: Here, we test whether TopP&R behaves stably under two variants of noise cases; 1) scatter noise : replacing X i and Y j with uniformly distributed noise and 2) swap noise : swapping the position between X i and Y j .
Revised: Here, we test whether TopP&R behaves stably under two variants of noise cases; 1) scatter noise : replacing X i and Y j with uniformly distributed noise and 2) swap noise : swapping the position between X i and Y j .

Original: These two cases all correspond to the adversarial noise model of Assumption 2.
Revised: These two cases all correspond to the adversarial noise model of Assumption 2.

Original: We set X ∼ N ( µ = 0 , I
Revised: We set X ∼ N ( µ = 0 , I

Original: ) ∈
Revised: ) ∈

Original: R 64 and Y ∼ N ( µ = 1 , I
Revised: R 64 and Y ∼ N ( µ = 1 , I

Original: ) ∈ R 64 where µ = 1 , and thus an ideal evaluation metric must return zero for both fidelity and diversity scores.
Revised: ) ∈ R 64 where µ = 1 , and thus an ideal evaluation metric must return zero for both fidelity and diversity scores.

Original: In both cases in Figure 4, we find that P&R and D&C are more sensitive while TopP&R remains relatively stable until the noise
Revised: In both cases in Figure 4, we find that P&R and D&C are more sensitive while TopP&R remains relatively stable until the noise

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ratio reaches 15% of the total data, which is a clear example of the weakness of existing metrics to perturbation.
Revised: ratio reaches 15% of the total data, which is a clear example of the weakness of existing metrics to perturbation.

Original: 5. S ANITY CHECK WITH R EAL DATA
Revised: 5. S ANITY CHECK WITH R EAL DATA

Original: Now that we have verified the metrics on toy data using Gaussians, we test them on real data.
Revised: Now that we have verified the metrics on toy data using Gaussians, we test them on real data.

Original: Just like in the toy experiments, we concentrate on how the metrics behave in extreme situations, such as outliers, mode-drop phenomena, perceptual distortions, and etc.
Revised: Just like in the toy experiments, we concentrate on how the metrics behave in extreme situations, such as outliers, mode-drop phenomena, perceptual distortions, and etc.

Original: For evaluation, ImageNet pretrained
Revised: For evaluation, ImageNet pretrained

Original: VGG16 with linear random projection to 32 dimension is used as an image embedding.
Revised: VGG16 with linear random projection to 32 dimension is used as an image embedding.

Original: For more experimental details, please refer to the appendix.
Revised: For more experimental details, please refer to the appendix.

Original: 5.2. R ESOLVING FIDELITY AND DIVERSITY
Revised: 5.2. R ESOLVING FIDELITY AND DIVERSITY

Original: To test whether TopP&R responds appropriately to the change in the underlying distributions in real scenarios, we test the metric on the generated images of stylegan2 (Karras et al., 2020) using the truncation trick (Karras et al., 2019).
Revised: To test whether TopP&R responds appropriately to the change in the underlying distributions in real scenarios, we test the metric on the generated images of stylegan2 (Karras et al., 2020) using the truncation trick (Karras et al., 2019).

Original: As shown in Figure 5, every time the distribution is transformed by ψ , TopP&R consistently responds with bounded scores in [0 , 1] .
Revised: As shown in Figure 5, every time the distribution is transformed by ψ , TopP&R consistently responds with bounded scores in [0 , 1] .

Original: On the other hand, Density gives unbounded scores (fidelity > 1).
Revised: On the other hand, Density gives unbounded scores (fidelity > 1).

Original: Because Density is not capped in value, it is difficult to interpret and know exactly which value denotes the best performance ( e.g. , in our case, the best performance is when fidelity and diversity = 1).
Revised: Because Density is not capped in value, it is difficult to interpret and know exactly which value denotes the best performance ( e.g. , in our case, the best performance is when fidelity and diversity = 1).

Original: 5.2. S EQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES IN CIFAR- 
Revised: 5.2. S EQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES IN CIFAR-

Original: We conduct an additional simultaneous mode drop experiment to verify TopP&R ’s actual sensitiveness on the real data set (CIFAR-10).
Revised: We conduct an additional simultaneous mode drop experiment to verify TopP&R ’s actual sensitiveness on the real data set (CIFAR-10).

Original: The performance of each metric (Figure A1) is measured with the identical data while simultaneously dropping the modes of nine classes of CIFAR10.
Revised: The performance of each metric (Figure A1) is measured with the identical data while simultaneously dropping the modes of nine classes of CIFAR10.

Original: Since the number of the images dropped in each step is identical, the trend of ground truth diversity should linearly decrease.
Revised: Since the number of the images dropped in each step is identical, the trend of ground truth diversity should linearly decrease.

Original: Here, P&R metric captures the simultaneous mode dropping better than D&C because this time random drop of the modes has reduced the area of the estimated fake manifold.
Revised: Here, P&R metric captures the simultaneous mode dropping better than D&C because this time random drop of the modes has reduced the area of the estimated fake manifold.

Original: On the other hand, TopP&R best captures the true trend of decreasing diversity on average, consistent with the toy result in Figure 3.
Revised: On the other hand, TopP&R best captures the true trend of decreasing diversity on average, consistent with the toy result in Figure 3.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: . R OBUSTNESS TO PERTURBATIONS BY OUTLYING FEATURES
Revised: 5.2. R OBUSTNESS TO PERTURBATIONS BY OUTLYING FEATURES

Original: To demonstrate the robustness of our metric against the adversarial noise model of Assumption 2, we test both scatter-noise and swap noise scenarios with real data.
Revised: To demonstrate the robustness of our metric against the adversarial noise model of Assumption 2, we test both scatter-noise and swap noise scenarios with real data.

Original: In the experiment, following
Revised: In the experiment, following

Original: Kynkäänniemi et al.
Revised: Kynkäänniemi et al.

Original: , we first classify inliers and outliers that are generated by StyleGAN (Karras et al., 2019).
Revised: , we first classify inliers and outliers that are generated by StyleGAN (Karras et al., 2019).

Original: For scatter noise we add the outliers to the inliers and for swap noise we swap the real FFHQ images with generated images.
Revised: For scatter noise we add the outliers to the inliers and for swap noise we swap the real FFHQ images with generated images.

Original: Under these specific noise conditions, Precision shows similar or even better robustness than Density (Figure 6).
Revised: Under these specific noise conditions, Precision shows similar or even better robustness than Density (Figure 6).

Original: On the other hand, Coverage is more robust than Recall .
Revised: On the other hand, Coverage is more robust than Recall .

Original: In both cases, TopP&R shows the best performance, resistant to noise.
Revised: In both cases, TopP&R shows the best performance, resistant to noise.

Original: 5.2. S ENSITIVENESS TO THE NOISE INTENSITY
Revised: 5.2. S ENSITIVENESS TO THE NOISE INTENSITY

Original: One of the advantages of FID (Heusel et al., 2017) is that it is good at estimating the degrees of distortion applied to the images.
Revised: One of the advantages of FID (Heusel et al., 2017) is that it is good at estimating the degrees of distortion applied to the images.

Original: Similarly, we check whether the F1-score based on TopP&R provides a reasonable evaluation according to different noise levels.
Revised: Similarly, we check whether the F1-score based on TopP&R provides a reasonable evaluation according to different noise levels.

Original: As illustrated in Figure 7, X and Y are sets of reference FFHQ features and noisy FFHQ features, respectively.
Revised: As illustrated in Figure 7, X and Y are sets of reference FFHQ features and noisy FFHQ features, respectively.

Original: The experimental results show that TopP&R actually reflects well the different degrees of distortion added to the images.
Revised: The experimental results show that TopP&R actually reflects well the different degrees of distortion added to the images.

Original: 5.2. R ANKING BETWEEN GENERATIVE MODELS
Revised: 5.2. R ANKING BETWEEN GENERATIVE MODELS

Original: One of the major caveats with two-score metrics is that they make it difficult to rank between different models; e.g. , which model is better?
Revised: One of the major caveats with two-score metrics is that they make it difficult to rank between different models; e.g. , which model is better?

Original: High fidelity with low diversity or low fidelity with high diversity?
Revised: High fidelity with low diversity or low fidelity with high diversity?

Original: In the case of traditional precision and Recall, this problem could be solved by using
Revised: In the case of traditional precision and Recall, this problem could be solved by using

Original: F1-score, which is the harmonic mean of fidelity and diversity.
Revised: F1-score, which is the harmonic mean of fidelity and diversity.

Original: However, unlike the traditional ones, the F1-score based on P&R or D&C does not provide a reliable or stable score due to their inherent instability and unboundedness.
Revised: However, unlike the traditional ones, the F1-score based on P&R or D&C does not provide a reliable or stable score due to their inherent instability and unboundedness.

Original: Thanks to its stability and robustness to various perturbations, we find that the TopP&R -based F1 score offers consistent ranking with FID (Table 1) 1 .
Revised: Thanks to its stability and robustness to various perturbations, we find that the TopP&R -based F1 score offers consistent ranking with FID (Table 1) 1 .

Original: [Table]
Revised: [Table]

Original:  R ELATED WORKS6. P ERSISTENT HOMOLOGY AND DEEP LEARNING
Revised:  R ELATED WORKS

Revised: Persistent homology and deep learning.

Original: Topology shows various potentials in the field of deep learning by introducing a new perspective on the support estimation, a new distance function robust to the noisy information, and a technique for GAN evaluation.
Revised: Topology shows various potentials in the field of deep learning by introducing a new perspective on the support estimation, a new distance function robust to the noisy information, and a technique for GAN evaluation.

Original: Chen et al.
Revised: Chen et al.

Original: (2017) introduces a method for approximating the support of a distribution using general density estimator and the Hausdorff distance and a new visualization method for support.
Revised: (2017) introduces a method for approximating the support of a distribution using general density estimator and the Hausdorff distance and a new visualization method for support.

Original: Chazal et al.
Revised: Chazal et al.

Original: (2011) proposes distance-to-measure, a robust Wasserstein distance function for perturbation, as an alternative to the characteristic that existing distance functions are not robust to outliers.
Revised: (2011) proposes distance-to-measure, a robust Wasserstein distance function for perturbation, as an alternative to the characteristic that existing distance functions are not robust to outliers.

Original: For the evaluation, one of the recent metric called MTop-Divergence (Barannikov et al., 2021) uses the summation (or in another word statistics) of the life-length of homology to score which manifold is containing more important topological signals.
Revised: For the evaluation, one of the recent metric called MTop-Divergence (Barannikov et al., 2021) uses the summation (or in another word statistics) of the life-length of homology to score which manifold is containing more important topological signals.

Original: While M-Top-Divergence directly use persistent homology to score the deep-learning models, we employ topology to estimate a robust and stable manifold.
Revised: While M-Top-Divergence directly use persistent homology to score the deep-learning models, we employ topology to estimate a robust and stable manifold.

Original: 6. E VALUATION METRICS
Revised: Evaluation metrics.

Original: Various evaluation metrics for generative models have been recently proposed (Salimans et al., 2016;
Revised: Various evaluation metrics for generative models have been recently proposed (Salimans et al., 2016;

Original: Heusel et al., 2017; Sajjadi et al., 2018; Kynkäänniemi et al., 2019; Naeem et al., 2020; Borji, 2022).
Revised: Heusel et al., 2017; Sajjadi et al., 2018; Kynkäänniemi et al., 2019; Naeem et al., 2020; Borji, 2022).

Original: One of the earliest methods is Inception Score (IS) (Szegedy et al., 2016), which measures the divergence of generated samples on the InceptionV3 embedding space.
Revised: One of the earliest methods is Inception Score (IS) (Szegedy et al., 2016), which measures the divergence of generated samples on the InceptionV3 embedding space.

Original: However, IS fails to capture the simultaneous mode drop and only considers the population distribution.
Revised: However, IS fails to capture the simultaneous mode drop and only considers the population distribution.

Original: Fréchet Inception Distance (FID) (Heusel et al., 2017) measures the difference in the means and variances of the real and fake features.
Revised: Fréchet Inception Distance (FID) (Heusel et al., 2017) measures the difference in the means and variances of the real and fake features.

Original: Since FID assumes the multi-Gaussian distribution of the features, if the true feature distribution is not normally distributed, the estimation becomes highly unreliable.
Revised: Since FID assumes the multi-Gaussian distribution of the features, if the true feature distribution is not normally distributed, the estimation becomes highly unreliable.

Original: Unlike IS and FID, which give a single score, some metrics separate the score into two components, the fidelity and diversity Sajjadi et al.
Revised: Unlike IS and FID, which give a single score, some metrics separate the score into two components, the fidelity and diversity Sajjadi et al.

Original: ; Kynkäänniemi et al.
Revised: ; Kynkäänniemi et al.

Original: ; Naeem et al.
Revised: ; Naeem et al.

Original: While Topological Precision and Recall ( TopP&R ) falls into this category, unlike the others, it does not assume strong regularity conditions.
Revised: While Topological Precision and Recall ( TopP&R ) falls into this category, unlike the others, it does not assume strong regularity conditions.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Recently, many works have been proposed to score the fidelity and diversity of generative models.
Revised: Recently, many works have been proposed to score the fidelity and diversity of generative models.

Original: However, none of them has focused on an accurate estimation of supports even though this is one of the key components in the evaluation pipeline.
Revised: However, none of them has focused on an accurate estimation of supports even though this is one of the key components in the evaluation pipeline.

Original: In this paper, we proposed topological precision and recall ( TopP&R ) that provides a systematical fix for robustly estimating the manifold by employing topological and statistical ideas.
Revised: In this paper, we proposed topological precision and recall ( TopP&R ) that provides a systematical fix for robustly estimating the manifold by employing topological and statistical ideas.

Original: Our theoretical and experimental results showed that TopP&R serves 
Revised: Our theoretical and experimental results showed that TopP&R serves as a robust and reliable evaluation metric under various noisy conditions, including mode collapse, outliers, and Non-IID perturbations.

Original: A PPENDIX A M ORE B ACKGROUND ON T OPOLOGICAL D ATA A NALYSIS
Revised: A PPENDIX A M ORE B ACKGROUND ON T OPOLOGICAL D ATA A NALYSIS

Original: Topological data analysis (TDA) (Carlsson, 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data.
Revised: Topological data analysis (TDA) (Carlsson, 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data.

Original: A key object in TDA is persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.
Revised: A key object in TDA is persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.

Original: Filtration.
Revised: Filtration.

Original: A filtration is a collection of subspaces approximating the data points at different resolutions, formally F = {F δ ⊂ R d } δ ∈ R such that δ 1 ≤ δ 2 implies that F δ 1 ⊂ F δ 2 .
Revised: A filtration is a collection of subspaces approximating the data points at different resolutions, formally F = {F δ ⊂ R d } δ ∈ R such that δ 1 ≤ δ 2 implies that F δ 1 ⊂ F δ 2 .

Original: Typically, a filtration is defined through a function f related to data.
Revised: Typically, a filtration is defined through a function f related to data.

Original: Given a function f : R d → R , we consider its sublevel filtration { f − 1 ( −∞ , δ ] } δ ∈ R or a superlevel filtration { f − 1 [ δ, ∞ ) } δ ∈ R .
Revised: Given a function f : R d → R , we consider its sublevel filtration { f − 1 ( −∞ , δ ] } δ ∈ R or a superlevel filtration { f − 1 [ δ, ∞ ) } δ ∈ R .

Original: Persistent homology.
Revised: Persistent homology.

Original: Persistent homology is a multiscale approach to represent the topological features, and is represented in the persistence diagram.
Revised: Persistent homology is a multiscale approach to represent the topological features, and is represented in the persistence diagram.

Original: For a filtration F and for each nonnegative k , we track when k -dimensional homological features (e.g., 0 -dimension: connected component, 1 -dimension: loop, 2 -dimension: cavity, . . .
Revised: For a filtration F and for each nonnegative k , we track when k -dimensional homological features (e.g., 0 -dimension: connected component, 1 -dimension: loop, 2 -dimension: cavity, . . .

Original: ) appear and disappear in the filtration.
Revised: ) appear and disappear in the filtration.

Original: As increasing or decreasing δ in the filtration {F δ } , if a homological feature appears at F b and disappears at F d , then we say that it is born at b and dies at d .
Revised: As increasing or decreasing δ in the filtration {F δ } , if a homological feature appears at F b and disappears at F d , then we say that it is born at b and dies at d .

Original: By considering these pairs { ( b, d ) } as points in the plane ( R ∪ {±∞} ) 2 , we obtain a persistence diagram .
Revised: By considering these pairs { ( b, d ) } as points in the plane ( R ∪ {±∞} ) 2 , we obtain a persistence diagram .

Original: From this, a homological feature with a longer life length, d − b , can be treated as a significant feature in the data set, and a homological feature with a shorter life length as a topological noise, which lies near the diagonal line { ( δ, δ ) : δ ∈ R } .
Revised: From this, a homological feature with a longer life length, d − b , can be treated as a significant feature in the data set, and a homological feature with a shorter life length as a topological noise, which lies near the diagonal line { ( δ, δ ) : δ ∈ R } .

Original: As discussed above, a homological feature with a long life-length is an important information in topology while the homology with short life-length can be treated as a non-significant information or noise.
Revised: As discussed above, a homological feature with a long life-length is an important information in topology while the homology with short life-length can be treated as a non-significant information or noise.

Original: The confidence band estimator provides the confidence set from the features that only includes topologically and statistically significant (statistically considered as elements in the population set) under a certain level of confidence.
Revised: The confidence band estimator provides the confidence set from the features that only includes topologically and statistically significant (statistically considered as elements in the population set) under a certain level of confidence.

Original: One way of constructing the confidence set uses the superlevel filtration of kernel density estimator and the bootstrap confidence band.
Revised: One way of constructing the confidence set uses the superlevel filtration of kernel density estimator and the bootstrap confidence band.

Original: Let X
Revised: Let X

Original: = { X 1 , X 2 , ..., X n } as given points cloud, then the probability for the distribution of points can be estimated via KDE defined as following: ˆ p h ( x ) :
Revised: = { X 1 , X 2 , ..., X n } as given points cloud, then the probability for the distribution of points can be estimated via KDE defined as following: ˆ p h ( x ) :

Original: = 1 nh d (cid:80) ni =1 K ( x − X
Revised: = 1 nh d (cid:80) ni =1 K ( x − X

Original: i
Revised: i

Original: h ) where h is the bandwidth and d as a dimension of the space.
Revised: h ) where h is the bandwidth and d as a dimension of the space.

Original: We derive estimated likelihood of X with KDE and likehood of ˜ p with using bootstrapped samples X ∗ .
Revised: We derive estimated likelihood of X with KDE and likehood of ˜ p with using bootstrapped samples X ∗ .

Original: Now, given the significance level α and h > 0 , let confidence band
Revised: Now, given the significance level α and h > 0 , let confidence band

Original: q X be bootstrap bandwidth of a Gaussian Empirical Process (van der Vaart, 2000; Kosorok, 2008), √ n ||
Revised: q X be bootstrap bandwidth of a Gaussian Empirical Process (van der Vaart, 2000; Kosorok, 2008), √ n ||

Original: ˆ p h − ˆ p ∗ h || ∞ .
Revised: ˆ p h − ˆ p ∗ h || ∞ .

Original: Then it satisfies P ( √ n || ˆ p
Revised: Then it satisfies P ( √ n || ˆ p

Original: h − p h
Revised: h − p h

Original: || ∞ <
Revised: || ∞ <

Original: q X ) ≥ 1 − α , as in Proposition 4 in Section C.
Revised: q X ) ≥ 1 − α , as in Proposition 4 in Section C.

Original: Then the ball of persistent homology centered at ˆ P h and radius c X = q X / √ n in the bottleneck distance d B is a valid confidence set as lim inf n →∞ P (cid:16) P ∈ B d B ( ˆ P h , c X ) (cid:17) ≥ 1 − α .
Revised: Then the ball of persistent homology centered at ˆ P h and radius c X = q X / √ n in the bottleneck distance d B is a valid confidence set as lim inf n →∞ P (cid:16) P ∈ B d B ( ˆ P h , c X ) (cid:17) ≥ 1 − α .

Original: This confidence set has further interpretation that in the persistence diagram, homological features that are above twice the radius 2 c X from the diagonal are simultaneously statistically significant.
Revised: This confidence set has further interpretation that in the persistence diagram, homological features that are above twice the radius 2 c X from the diagonal are simultaneously statistically significant.

Original: B D ENOISING TOPOLOGICAL FEATURES FROM OUTLIERS
Revised: B D ENOISING TOPOLOGICAL FEATURES FROM OUTLIERS

Original: Using the bootstrap bandwidth c X as threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) .
Revised: Using the bootstrap bandwidth c X as threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) .

Original: When the level set ˆ p − 1 h [ c X , ∞ ) is used, the homology of ˆ p − 1 h [ c X , ∞ ) consists of homological features whose (birth) ≥ c X and ( death ) ≤ c X , which are the homological features in skyblue area in Figure 1.
Revised: When the level set ˆ p − 1 h [ c X , ∞ ) is used, the homology of ˆ p − 1 h [ c X , ∞ ) consists of homological features whose (birth) ≥ c X and ( death ) ≤ c X , which are the homological features in skyblue area in Figure 1.

Original: Particularly, the homological features generated by outliers have their (birth) <
Revised: Particularly, the homological features generated by outliers have their (birth) <

Original: c X simultaneously with high probability, so those homological noise are removed in the estimated support ˆ p − 1 h [ c X , ∞ ) .
Revised: c X simultaneously with high probability, so those homological noise are removed in the estimated support ˆ p − 1 h [ c X , ∞ ) .

Original: C A SSUMPTIONS ON DISTRIBUTIONS AND KERNELS
Revised: C A SSUMPTIONS ON DISTRIBUTIONS AND KERNELS

Original: Assumption A1.
Revised: Assumption A1.

Original: Let K : R d → R be a nonnegative function with ∥ K ∥ 1 = 1 , ∥ K ∥ ∞ , ∥ K
Revised: Let K : R d → R be a nonnegative function with ∥ K ∥ 1 = 1 , ∥ K ∥ ∞ , ∥ K

Original: ∥
Revised: ∥

Original: 2 < ∞ , and satisfy the following
Revised: 2 < ∞ , and satisfy the following

Original: : (1) K (0) > 0 .
Revised: : (1) K (0) > 0 .

Original: (2) K has a compact support.
Revised: (2) K has a compact support.

Original: (3) K is Lipschitz continuous and of second order.
Revised: (3) K is Lipschitz continuous and of second order.

Original: Assumption A1, allows to build a valid bootstrap confidence band for kernel density estimator (KDE).
Revised: Assumption A1, allows to build a valid bootstrap confidence band for kernel density estimator (KDE).

Original: See Theorem 12 of (Fasy et al., 2014) or Theorem 3.4 of (Neumann, 1998)
Revised: See Theorem 12 of (Fasy et al., 2014) or Theorem 3.4 of (Neumann, 1998)

Original: Proposition 4 (Theorem 3.4 of (Neumann, 1998)) .
Revised: Proposition 4 (Theorem 3.4 of (Neumann, 1998)) .

Original: Let X
Revised: Let X

Original: =
Revised: =

Original: X 1 , . .
Revised: X 1 , . .

Original: X n } be IID from a distribution
Revised: X n } be IID from a distribution

Original: P .
Revised: P .

Original: For h > 0 , let ˆ p h , ˆ p ∗ h be kernel density estimator for X and its bootstrap X ∗ , respectrively, and for α ∈ (0 , 1) , let c
Revised: For h > 0 , let ˆ p h , ˆ p ∗ h be kernel density estimator for X and its bootstrap X ∗ , respectrively, and for α ∈ (0 , 1) , let c

Original: X
Revised: X

Original: be the α bootstrap quantile from √ nh d ∥ ˆ p h − ˆ p ∗ h ∥ ∞ .
Revised: be the α bootstrap quantile from √ nh d ∥ ˆ p h − ˆ p ∗ h ∥ ∞ .

Original: For h n → 0 ,
Revised: For h n → 0 ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 3, A1 ensures that, when the bandwidth h n → 0 , average KDEs are bounded away from 0 .
Revised: Assumption 3, A1 ensures that, when the bandwidth h n → 0 , average KDEs are bounded away from 0 .

Original: Lemma 5.
Revised: Lemma 5.

Original: Let P be a distribution satisfying Assumption 3.
Revised: Let P be a distribution satisfying Assumption 3.

Original: Suppose K is a nonnegative function satisfying K (0) > 0 and continuous at 0 .
Revised: Suppose K is a nonnegative function satisfying K (0) > 0 and continuous at 0 .

Original: Suppose { h n } n ∈ N with h n ≥ 0 and h n → 0 is given.
Revised: Suppose { h n } n ∈ N with h n ≥ 0 and h n → 0 is given.

Original: Then for all x ∈ supp( P ) , lim inf
Revised: Then for all x ∈ supp( P ) , lim inf

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since K (0) > 0 and K is continuous at 0 , there is r 0 > 0 such that for all y ∈ B (0 , r 0 ) ,
Revised: Since K (0) > 0 and K is continuous at 0 , there is r 0 > 0 such that for all y ∈ B (0 , r 0 ) ,

Original: K ( y ) ≥ 12 K (0) > 0 .
Revised: K ( y ) ≥ 12 K (0) > 0 .

Original: And hence
Revised: And hence

Original: [Equation]
Revised: [Equation]

Original: d P ( B ( x, r 0 h )) .
Revised: d P ( B ( x, r 0 h )) .

Original: Hence as h n → 0 ,
Revised: Hence as h n → 0 ,

Original: lim inf n p h
Revised: lim inf n p h

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: >
Revised: >

Original: D D ETAILS AND P ROOFS FOR S ECTION
Revised: D D ETAILS AND P ROOFS FOR S ECTION

Original: Let ˜ p
Revised: Let ˜ p

Original: h be the KDE on X 0 .
Revised: h be the KDE on X 0 .

Original: For a finite set X , we use the notation c X ,α for α -bootstrap quantile satisfying P (cid:0)(cid:13)(cid:13)
Revised: For a finite set X , we use the notation c X ,α for α -bootstrap quantile satisfying P (cid:0)(cid:13)(cid:13)

Original: ˆ p X ,h n − ˆ p X b ,h n (cid:13)(cid:13) ∞ >
Revised: ˆ p X ,h n − ˆ p X b ,h n (cid:13)(cid:13) ∞ >

Original: c X ,α |X (cid:1)
Revised: c X ,α |X (cid:1)

Original: = 1 − α , where X b is the bootstrap sample from X .
Revised: = 1 − α , where X b is the bootstrap sample from X .

Original: For a distribution P , we use the notation c P,α for α -quantile satisfying P ( ∥ ˆ p h n − p h n ∥ ∞ > c P,α )
Revised: For a distribution P , we use the notation c P,α for α -quantile satisfying P ( ∥ ˆ p h n − p h n ∥ ∞ > c P,α )

Original: = 1 − α , where ˆ p h n is kernel density estimator of IID samples from P .
Revised: = 1 − α , where ˆ p h n is kernel density estimator of IID samples from P .

Original: Hence when X is not IID samples from P , the relation of Proposition 4 may not hold.
Revised: Hence when X is not IID samples from P , the relation of Proposition 4 may not hold.

Original: Lemma 6.
Revised: Lemma 6.

Original: Under Assumption 1, 2 and A1,
Revised: Under Assumption 1, 2 and A1,

Original: [Equation]
Revised: [Equation]

Original: (2) Under Assumption 1, 2 and A1,
Revised: (2) Under Assumption 1, 2 and A1,

Original: [Equation]
Revised: [Equation]

Original: (3) Suppose Assumption 1, 2,A1 hold, and suppose nh − dn ρ 2 n → 0 .
Revised: (3) Suppose Assumption 1, 2,A1 hold, and suppose nh − dn ρ 2 n → 0 .

Original: Then with probability 1 −
Revised: Then with probability 1 −

Original: [Equation]
Revised: [Equation]

Original: 2 δ ,
Revised: 2 δ ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, note that
Revised: First, note that

Original: [Equation]
Revised: [Equation]

Original: Then under Assumption A1,
Revised: Then under Assumption A1,

Original: [Equation]
Revised: [Equation]

Original: Then from Assumption 2, (cid:80) ni
Revised: Then from Assumption 2, (cid:80) ni

Original: =

Revised: =

Original: I (cid:0) X i ̸
Revised: I (cid:0) X i ̸

Original: =
Revised: =

Original: X 

Revised: X

Original: i (cid:1) ≤ nρ n , and hence
Revised: i (cid:1) ≤ nρ n , and hence

Original: [Equation]
Revised: [Equation]

Original: Let X b , X 0 b be bootstrapped samples of X , X 0 with the same sampling with replacement process.
Revised: Let X b , X 0 b be bootstrapped samples of X , X 0 with the same sampling with replacement process.

Original: Let ˆ p bh , ˜ p
Revised: Let ˆ p bh , ˜ p

Original: bh be KDE of X b and X 0 b , respectively.
Revised: bh be KDE of X b and X 0 b , respectively.

Original: And, note that
Revised: And, note that

Original: [Equation]
Revised: [Equation]

Original: Let L b be the number of elements where X b and X 0 b differ, i.e., L b = (cid:12)(cid:12)
Revised: Let L b be the number of elements where X b and X 0 b differ, i.e., L b = (cid:12)(cid:12)

Original: X b \X 0 b (cid:12)(cid:12) = (cid:12)(cid:12)
Revised: X b \X 0 b (cid:12)(cid:12) = (cid:12)(cid:12)

Original: X 

Revised: X

Original: b \X b (cid:12)(cid:12) , then
Revised: b \X b (cid:12)(cid:12) , then

Original: L b ∼ Binomial( n, ρ n ) , and (cid:13)(cid:13)
Revised: L b ∼ Binomial( n, ρ n ) , and (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: And hence,
Revised: And hence,

Original: [Equation]
Revised: [Equation]

Original: Then by using subgaussian tail bound, with probability 1 − δ ,
Revised: Then by using subgaussian tail bound, with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence this implies
Revised: Hence this implies

Original: [Equation]
Revised: [Equation]

Original: Since X 0 is IID samples from P , with probability 1 − α − 2 δ ,
Revised: Since X 0 is IID samples from P , with probability 1 − α − 2 δ ,

Original: [Equation]
Revised: [Equation]

Original: δ + O (cid:18)nh d (cid:19) .
Revised: δ + O (cid:18)nh d (cid:19) .

Original: Now, note that c X 0 ,α = Θ (cid:18)(cid:113) log(1 /α )
Revised: Now, note that c X 0 ,α = Θ (cid:18)(cid:113) log(1 /α )

Original: nh d (cid:19) , and hence
Revised: nh d (cid:19) , and hence

Original: [Equation]
Revised: [Equation]

Original: Then under Assumption 2, since nh − dn ρ 2 n = o (1) and h − dn ρ n = o (
Revised: Then under Assumption 2, since nh − dn ρ 2 n = o (1) and h − dn ρ n = o (

Original: [Equation]
Revised: [Equation]

Original: ≤ c
Revised: ≤ c

Original: [Equation]
Revised: [Equation]

Original: α −
Revised: α −

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: Corollary 7.
Revised: Corollary 7.

Original: Suppose Assumption 1, 2, A1 hold.
Revised: Suppose Assumption 1, 2, A1 hold.

Original: (1) Suppose nh − dn ρ 2 n → 0 .
Revised: (1) Suppose nh − dn ρ 2 n → 0 .

Original: Then with probability 1 − α − 2 δ , p − 1 h n [2 c P,α − 2 δ , ∞ ) ⊂
Revised: Then with probability 1 − α − 2 δ , p − 1 h n [2 c P,α − 2 δ , ∞ ) ⊂

Original: ˆ p − 1 h n [ c X ,α , ∞ ) ⊂ supp( P h n ) .
Revised: ˆ p − 1 h n [ c X ,α , ∞ ) ⊂ supp( P h n ) .

Original: (2) Suppose mh − dm ρ 2 m → 0 .
Revised: (2) Suppose mh − dm ρ 2 m → 0 .

Original: Then with probability 1 − α − 2 δ , q
Revised: Then with probability 1 − α − 2 δ , q

Original: [Equation]
Revised: [Equation]

Original: 1 h m
Revised: 1 h m

Original: [Equation]
Revised: [Equation]

Original: 2 c Q,α − 2 δ , ∞ ) ⊂
Revised: 2 c Q,α − 2 δ , ∞ ) ⊂

Original: ˆ q − 1 h m [ c Y ,α , ∞ )
Revised: ˆ q − 1 h m [ c Y ,α , ∞ )

Original: ⊂ supp( Q h m ) .
Revised: ⊂ supp( Q h m ) .

Original: Proof.
Revised: Proof.

Original: (1) From Lemma 6, ∥ ˆ p h − p h
Revised: (1) From Lemma 6, ∥ ˆ p h − p h

Original: ∥ <
Revised: ∥ <

Original: c X ,α ≤ c P,α − 2 δ .
Revised: c X ,α ≤ c P,α − 2 δ .

Original: This implies p − 1 h n [2 c P,α − 2 δ , ∞ ) ⊂ ˆ p − 1 h n [ c X ,α , ∞ )
Revised: This implies p − 1 h n [2 c P,α − 2 δ , ∞ ) ⊂ ˆ p − 1 h n [ c X ,α , ∞ )

Original: ⊂ supp( P h n ) .
Revised: ⊂ supp( P h n ) .

Original: (2) can be proven similarly to (1).
Revised: (2) can be proven similarly to (1).

Original: Claim 8 .
Revised: Claim 8 .

Original: For a nonnegative measure µ and sets A, B, C, D, µ ( A ∩ B ) − µ ( C ∩ D ) ≤ µ ( A \ C ) + µ ( B \ D ) .
Revised: For a nonnegative measure µ and sets A, B, C, D, µ ( A ∩ B ) − µ ( C ∩ D ) ≤ µ ( A \ C ) + µ ( B \ D ) .

Original: Proof.
Revised: Proof.

Original: µ (
Revised: µ (

Original: [Equation]
Revised: [Equation]

Original: ) − µ ( C ∩ D ) ≤ µ (( A ∩ B ) \ ( C ∩ D )) = µ (( A ∩ B ) ∩ ( C ∁ ∪ D ∁ ))
Revised: ) − µ ( C ∩ D ) ≤ µ (( A ∩ B ) \ ( C ∩ D )) = µ (( A ∩ B ) ∩ ( C ∁ ∪ D ∁ ))

Original: = µ (( A ∩ B ) ∩ C ∁ ) ∪ ( A ∩ B ) ∩ D ∁ )) ≤ µ (( A ∩ B ) \ C ) + µ ( A ∩ B ) \ D ) ≤ µ ( A \ C ) + µ ( B \ D ) .
Revised: = µ (( A ∩ B ) ∩ C ∁ ) ∪ ( A ∩ B ) ∩ D ∁ )) ≤ µ (( A ∩ B ) \ C ) + µ ( A ∩ B ) \ D ) ≤ µ ( A \ C ) + µ ( B \ D ) .

Original: From here, let P n and Q m be the empirical measures on X and Y ,
Revised: From here, let P n and Q m be the empirical measures on X and Y ,

Original: respectively, i.e., P n = 1 n (cid:80) ni
Revised: respectively, i.e., P n = 1 n (cid:80) ni

Original: =

Revised: =

Original: δ X i and Q m = 1 m (cid:80) mj
Revised: δ X i and Q m = 1 m (cid:80) mj

Original: =1 δ Y j .
Revised: =1 δ Y j .

Original: Lemma 9.
Revised: Lemma 9.

Original: Suppose Assumption 1, 2 hold.
Revised: Suppose Assumption 1, 2 hold.

Original: (1) Let A ⊂ R d .
Revised: (1) Let A ⊂ R d .

Original: Then with probability 1 − δ , | P n − P | ( A ) =
Revised: Then with probability 1 − δ , | P n − P | ( A ) =

Original: o (1) .
Revised: o (1) .

Original: (2) Let A n ⊂ R d be a sequence of set satisfying A n → ∅ , i.e., lim sup n A n = ∅ .
Revised: (2) Let A n ⊂ R d be a sequence of set satisfying A n → ∅ , i.e., lim sup n A n = ∅ .

Original: Then P n ( A n ) → 0 in probability .
Revised: Then P n ( A n ) → 0 in probability .

Original: Proof.
Revised: Proof.

Original: Since P ( A n ) ≤ P ( (cid:83) ni =1 A i ) and (cid:83)
Revised: Since P ( A n ) ≤ P ( (cid:83) ni =1 A i ) and (cid:83)

Original: ni
Revised: ni

Original: =

Revised: =

Original: A i → ∅ as well, we can assume that A n ↓ ∅ , i.e.,
Revised: A i → ∅ as well, we can assume that A n ↓ ∅ , i.e.,

Original: A n ⊃ A n +1 for all n and (cid:84) ∞ n =1 A n = ∅ .
Revised: A n ⊃ A n +1 for all n and (cid:84) ∞ n =1 A n = ∅ .

Original: Let P 0 n be the empirical measure on X 0 , i.e., P 0 n = 1 n (cid:80) ni
Revised: Let P 0 n be the empirical measure on X 0 , i.e., P 0 n = 1 n (cid:80) ni

Original: =1 δ X i .
Revised: =1 δ X i .

Original: By using subgaussian tail bound, with probability 1 − δ , (cid:12)(cid:12) P
Revised: By using subgaussian tail bound, with probability 1 − δ , (cid:12)(cid:12) P

Original: [Equation]
Revised: [Equation]

Original: n − P (cid:12)(cid:12) ( A ) = O (cid:32)(cid:114) log(1 /δ )
Revised: n − P (cid:12)(cid:12) ( A ) = O (cid:32)(cid:114) log(1 /δ )

Original: n (cid:33) .
Revised: n (cid:33) .

Original: And (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12)
Revised: And (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12)

Original: ( A ) is expanded as (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12) ( A ) = 1 n n (cid:88) i = (cid:12)(cid:12)
Revised: ( A ) is expanded as (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12) ( A ) = 1 n n (cid:88) i = (cid:12)(cid:12)

Original: I ( X
Revised: I ( X

Original: i ∈
Revised: i ∈

Original: A )
Revised: A )

Original: − I ( X 0 i ∈ A ) (cid:12)(cid:12) .
Revised: − I ( X 0 i ∈ A ) (cid:12)(cid:12) .

Original: Under Assumption 2 , (cid:80) ni =1 I (cid:0) X i ̸
Revised: Under Assumption 2 , (cid:80) ni =1 I (cid:0) X i ̸

Original: =
Revised: =

Original: X 

Revised: X

Original: i (cid:1) ≤ nρ n , and hence (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12) ( A ) = 1 n n (cid:88) i = (cid:12)(cid:12) I
Revised: i (cid:1) ≤ nρ n , and hence (cid:12)(cid:12) P n − P 0 n (cid:12)(cid:12) ( A ) = 1 n n (cid:88) i = (cid:12)(cid:12) I

Original: ( X i ∈ A ) − I
Revised: ( X i ∈ A ) − I

Original: ( X 0 i ∈ A ) (cid:12)(cid:12) ≤n n (cid:88) i =
Revised: ( X 0 i ∈ A ) (cid:12)(cid:12) ≤n n (cid:88) i =

Original: I (cid:0) X i ̸
Revised: I (cid:0) X i ̸

Original: =
Revised: =

Original: X 0 i (cid:1) =
Revised: X 0 i (cid:1) =

Original: ρ n .
Revised: ρ n .

Original: Therefore, with probability 1 − δ , | P n − P | ( A ) ≤ O (cid:32) ρ n +
Revised: Therefore, with probability 1 − δ , | P n − P | ( A ) ≤ O (cid:32) ρ n +

Original: [Equation]
Revised: [Equation]

Original: (2) Note that P n ( A n ) can be bounded as
Revised: (2) Note that P n ( A n ) can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Fix δ > 0 and ϵ > 0 .
Revised: Fix δ > 0 and ϵ > 0 .

Original: Since lim n P ( A n ) = 0 , we can choose N 1 such that P
Revised: Since lim n P ( A n ) = 0 , we can choose N 1 such that P

Original: ( A N 1 )
Revised: ( A N 1 )

Original: <
Revised: <

Original: ϵ , and we can choose N 2 such that for all n ≥ N 2 , P ( | P n − P | ( A N 1 ) < ϵ ) ≥ 1 − δ .
Revised: ϵ , and we can choose N 2 such that for all n ≥ N 2 , P ( | P n − P | ( A N 1 ) < ϵ ) ≥ 1 − δ .

Original: Then for all n ≥ max { N 1 , N 2 } , with probability 1 − δ ,
Revised: Then for all n ≥ max { N 1 , N 2 } , with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence P n ( A n ) → 0 in probability.
Revised: Hence P n ( A n ) → 0 in probability.

Original: Claim 10 .
Revised: Claim 10 .

Original: Suppose Assumption 1, 2, 3, A1 hold.
Revised: Suppose Assumption 1, 2, 3, A1 hold.

Original: [Equation]
Revised: [Equation]

Original: in probability .
Revised: in probability .

Original: [Equation]
Revised: [Equation]

Original: supp
Revised: supp

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: → 0 in probability .
Revised: → 0 in probability .

Original: (3) Q m (cid:0) ˆ p − 1 h n [ c X , ∞ ) ∩ ˆ q − 1 h m [ c Y , ∞ ) (cid:1) → Q (supp( P )) in probability .
Revised: (3) Q m (cid:0) ˆ p − 1 h n [ c X , ∞ ) ∩ ˆ q − 1 h m [ c Y , ∞ ) (cid:1) → Q (supp( P )) in probability .

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: 1 h
Revised: 1 h

Original: [Equation]
Revised: [Equation]

Original: Y
Revised: Y

Original: [Equation]
Revised: [Equation]

Original: ∞
Revised: ∞

Original: [Equation]
Revised: [Equation]

Original: → 1 in probability .
Revised: → 1 in probability .

Original: Proof.
Revised: Proof.

Original: From Lemma 7, with high probability,
Revised: From Lemma 7, with high probability,

Original: [Equation]
Revised: [Equation]

Original: Then from the first inequality, combining with Claim 8 gives
Revised: Then from the first inequality, combining with Claim 8 gives

Original: [Equation]
Revised: [Equation]

Original: And from the second inequality, combining with Claim 8 gives
Revised: And from the second inequality, combining with Claim 8 gives

Original: [Equation]
Revised: [Equation]

Original: And hence
Revised: And hence

Original: [Equation]
Revised: [Equation]

Original: Now, note that from Lemma 5 implies that for all x ∈ supp( P ) , lim inf n p h n ( x ) > 0 , so p h n ( x ) > 2 c P for large enough n .
Revised: Now, note that from Lemma 5 implies that for all x ∈ supp( P ) , lim inf n p h n ( x ) > 0 , so p h n ( x ) > 2 c P for large enough n .

Original: And hence supp( P
Revised: And hence supp( P

Original: [Equation]
Revised: [Equation]

Original: \ p
Revised: \ p

Original: [Equation]
Revised: [Equation]

Original: 1 h n [2 c P , ∞ ) → ∅ .
Revised: 1 h n [2 c P , ∞ ) → ∅ .

Original: And similar argument holds for supp( Q ) \ q − 1 h m [2 c Q , ∞ ) , so supp( Q ) \ q − 1 h m [2 c Q , ∞ )
Revised: And similar argument holds for supp( Q ) \ q − 1 h m [2 c Q , ∞ ) , so supp( Q ) \ q − 1 h m [2 c Q , ∞ )

Original: → ∅ as well.
Revised: → ∅ as well.

Original: Then from Lemma 9,
Revised: Then from Lemma 9,

Original: [Equation]
Revised: [Equation]

Original: Also, since K has compact support, for any x / ∈ supp( P ) , x / ∈ supp( P h n ) once h n < d ( x, supp( P )) .
Revised: Also, since K has compact support, for any x / ∈ supp( P ) , x / ∈ supp( P h n ) once h n < d ( x, supp( P )) .

Original: Hence supp( P h n ) \ supp( P ) → ∅ , and similarly supp( Q h m ) \ supp( Q ) → ∅ as well.
Revised: Hence supp( P h n ) \ supp( P ) → ∅ , and similarly supp( Q h m ) \ supp( Q ) → ∅ as well.

Original: Then again with Lemma 9,
Revised: Then again with Lemma 9,

Original: [Equation]
Revised: [Equation]

Original: And hence
Revised: And hence

Original: [Equation]
Revised: [Equation]

Original: This can be done similarly to (1).
Revised: This can be done similarly to (1).

Original: Lemma 9 (1) gives that with probability 1 − δ ,
Revised: Lemma 9 (1) gives that with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence combining with (1) gives the desired result.
Revised: Hence combining with (1) gives the desired result.

Original: Lemma 9 (1) gives that with probability 1 − δ ,
Revised: Lemma 9 (1) gives that with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Hence combining with (2) gives the desired result.
Revised: Hence combining with (2) gives the desired result.

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: Now this is a combination of Claim 10 (1) (2).
Revised: Now this is a combination of Claim 10 (1) (2).

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: Now this is a combination of Claim 10 (3) (4).
Revised: Now this is a combination of Claim 10 (3) (4).

Original: E E XPERIMENTAL D ETAILS E. C HOICE OF CONFIDENCE LEVEL

Original: For the confidence level α , we would like to point out that α is not the usual hyperparameter to be tuned: It has a statistical interpretation of the probability or the level of confidence to allow error, noise, etc.

Original: The most popular choices are α = 0 .

Original: 05 , 0 . 01 , leading to 90%, 95%, 99% confidence.

Original: We used α = 0 .

Original: 1 throughout our experiments.

Original: E. E STIMATION OF B ANDWIDTH PARAMETER

Original: As we discussed in section 2, since TopP&R estimates the manifold through KDE with kernel bandwidth parameter h , we need to approximate it.

Original: The estimation techniques for h are as follows: (1) a method of selecting h that maximizes the survival time ( S ( h ) ) or the number of significant homological features ( N ( h ) ) based on information obtained about persistent homology using the filtration method, (2) a method using the median of the k-nearest neighboring distances between features obtained by the balloon estimator (for more details, please refer to Wagner et al.

Original: (2012) and

Original: Terrell & Scott (1992)).

Original: Note that, the bandwidth h for all the experiments in this paper are estimated via Balloon Bandwidth Estimator.

Original: Algorithm 2 Balloon Bandwidth Estimator 1: # h : bandwidth; KND: kth nearest distance; idx: index 2: Given X = { X 1 , X 2 , . . .

Original: , X n } 3: for idx = 1 , 2 , . . .

Original: , n do 4: # Compute L2 distance

Original: X idx and X 5: d

Original: [Equation]

Original: , 6: i.e. d

Original: [Equation]

Original: 7: # Define the kth nearest neighbor distance by sorting in ascending order 8:

Original: KND

Original: [Equation]

Original: [k] 9: end for 10:

Original: Given kth nearest distance set: KND = { KND 1 , KND 2 , . . .

Original: , KND n } 11: # Define the estimated bandwidth ˆ h 12: ˆ h = median ( KND )

Original: E. E XPLICIT VALUES OF BANDWIDTH PARAMETER
Revised: E E XPERIMENTAL D ETAILS E. I MPLEMENTATION DETAILS OF EMBEDDING

Original: [Table] [Table] E. I MPLEMENTATION DETAILS OF EMBEDDING .

Original: We summarize the detailed information of our embedding networks implemented for the experiments.
Revised: We summarize the detailed information of our embedding networks implemented for the experiments.

Original: In Figure 2, 3, 4, 5, A

Original: [Table]

Original: and

Original: [Table]

Original: P&R and D&C are computed from the features mapped by an ImageNet

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] -trained VGG16, and TopP&R is computed from features placed in R 32 with additional random linear projection.
Revised: In Figure 2, 3, 4, 5, A1, and 6, P&R and D&C are computed from the features mapped by an ImageNet pre-trained VGG16, and TopP&R is computed from features placed in R 32 with additional random linear projection.

Original: In the experiment in Figure 5, the SwAV embedder is additionally considered.
Revised: In the experiment in Figure 5, the SwAV embedder is additionally considered.

Original: We implement ImageNet pre-trained InceptionV3, VGG16, and SwAV as embedding networks to compare the ranking of GANs in Table 1.
Revised: We implement ImageNet pre-trained InceptionV3, VGG16, and SwAV as embedding networks to compare the ranking of GANs in Table 1.

Original: The random projection is characterized by preserving the information about distances and homological features defined in the VGG16 feature space by
Revised: The random projection is characterized by preserving the information about distances and homological features defined in the VGG16 feature space by

Original: Johnson Linenstrauss Lemma (Johnson et al., 1986).
Revised: Johnson Linenstrauss Lemma (Johnson et al., 1986).

Original: [Table]

Original: [Table]

Original: E. M EAN HAMMING DISTANCE.
Revised: E. M EAN HAMMING DISTANCE

Original: Hamming distance (HD) (Hamming, 1950) is a metric that counts the number of items with different ranks in A and B and measures how much proportion differs in the overall order, i.e. for A i ∈ A and B i ∈ B ,
Revised: Hamming distance (HD) (Hamming, 1950) is a metric that counts the number of items with different ranks in A and B and measures how much proportion differs in the overall order, i.e. for A i ∈ A and B i ∈ B ,

Original: HD ( A, B ) :
Revised: HD ( A, B ) :

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 1 { k : A i ̸
Revised: =1 1 { k : A i ̸

Original: = B i } /n where n is the number of items in list A or
Revised: = B i } /n where n is the number of items in list A or

Original: B, and k is the number of differently ordered elements.
Revised: B, and k is the number of differently ordered elements.

Original: The mean HD is calculated as follows to measure the average distances of three ordered lists: Given three ordered lists A, B, and C , ¯
Revised: The mean HD is calculated as follows to measure the average distances of three ordered lists: Given three ordered lists A, B, and C , ¯

Original: HD = ( HD ( A, B ) + HD ( A, C )
Revised: HD = ( HD ( A, B ) + HD ( A, C )

Original: + HD ( B, C )) /
Revised: + HD ( B, C )) /

Revised: E.

Revised: C HOICE OF CONFIDENCE LEVEL

Revised: For the confidence level α , we would like to point out that α is not the usual hyperparameter to be tuned: It has a statistical interpretation of the probability or the level of confidence to allow error, noise, etc.

Revised: The most popular choices are α = 0 .

Original: F A DDITIONAL E XPERIMENTS F. SEQUENTIAL AND SIMULTANEOUS MODE DROPPING WITH CIFAR-
Revised: 05 , 0 . 01 , leading to 90%, 95%, 99% confidence. We used α = 0 . 1 throughout our experiments. E. E STIMATION OF B ANDWIDTH PARAMETER

Revised: As we discussed in section 2, since TopP&R estimates the manifold through KDE with kernel bandwidth parameter h , we need to approximate it.

Revised: The estimation techniques for h are as follows: (1) a method of selecting h that maximizes the survival time ( S ( h ) ) or the number of significant homological features ( N ( h ) ) based on information obtained about persistent homology using the filtration method, (2) a method using the median of the k-nearest neighboring distances between features obtained by the balloon estimator (for more details, please refer to Wagner et al.

Revised: (2012) and

Revised: Terrell & Scott (1992)).

Revised: Note that, the bandwidth h for all the experiments in this paper are estimated via Balloon Bandwidth Estimator.

Revised: 9: end

Revised: E. E XPLICIT VALUES OF BANDWIDTH PARAMETER h

Revised: Since our metric adaptively reacts to the given samples of P and Q

Revised: [Table]

Revised: we have two h s per experiment.

Revised: For example, in the translation experiment (Figure 2)

Revised: [Table]

Revised: there are 13 steps in total

Original: [Figure]
Revised: [Table]

Revised: and each time we estimate h for P and Q , resulting in a total of 26 h s. To show them all at a glance, we have listed all values in one place.

Revised: We will also provide the code that can reproduce the results in our experiments upon acceptance.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  F. S URVIVABILITY OF MINORITY SETS IN THE LONG - TAILED DISTRIBUTION
Revised: [Table] [Table] [Table] F A DDITIONAL E XPERIMENTS F. S URVIVABILITY OF MINORITY SETS IN THE LONG - TAILED DISTRIBUTION

Revised: [Table]

Original: An important point to check in our proposed metric is the possibility that a small part of the total data, but containing important information, can be ignored by the confidence band.
Revised: An important point to check in our proposed metric is the possibility that a small part of the total data, but containing important information, can be ignored by the confidence band.

Original: To test this, we experimented with CIFAR10, which has 5,000 samples per class.
Revised: To test this, we experimented with CIFAR10, which has 5,000 samples per class.

Original: We simulate a dataset with the majority set of six classes (2,000 samples per class, 12,000 total) and the minority set of four classes (500 samples per class, 2,000 total), and an ideal generator that exactly mimics the full data[Table] distribution.
Revised: We simulate a dataset with the majority set of six classes (2,000 samples per class, 12,000 total) and the minority set of four classes (500 samples per class, 2,000 total), and an ideal generator that exactly mimics the full data distribution.

Original: As shown in the Table 12, the samples in the minority set remained after the filtering process, meaning that the samples were sufficient to form a significant structure.
Revised: As shown in the Table 12, the samples in the minority set remained after the filtering process, meaning that the samples were sufficient to form a significant structure.

Original: Both D&C and
Revised: Both D&C and

Original: TopP&R successfully evaluate the distribution for the ideal generator.
Revised: TopP&R successfully evaluate the distribution for the ideal generator.

Original: To check whether our metric reacts to the change in the distribution even with this harsh setting, we also carried out the mode decay experiment.
Revised: To check whether our metric reacts to the change in the distribution even with this harsh setting, we also carried out the mode decay experiment.

Original: We dropped the samples of the minority set from 500 to 100 per class, which can be interpreted as an 11.3% decrease in diversity relative to the full distribution (Given (1) ratio of the number of samples between majority and minority sets = 12 , 000 : 2 , 000 = 6 : 1 and (2) 80% decrease in samples per minority class, the true decay in the diversity is calculated as[Equation] [Equation]
Revised: We dropped the samples of the minority set from 500 to 100 per class, which can be interpreted as an 11.3% decrease in diversity relative to the full distribution (Given (1) ratio of the number of samples between majority and minority sets = 12 , 000 : 2 , 000 = 6 : 1 and (2) 80% decrease in samples per minority class, the true decay in the diversity is calculated as

Original: [Equation]
Revised: [Figure]

Original: with respect to the enitre samples).
Revised: with respect to the enitre samples).

Original: Here, recall and coverage react somewhat less sensitively with their reduced diversities as 3 p.p. and 2 p.p., respectively, while TopR reacted most similarly (9 p.p.) to the ideal value. 
Revised: Here, recall and coverage react somewhat less sensitively with their reduced diversities as 3 p.p. and 2 p.p., respectively, while TopR reacted most similarly (9 p.p.) to the ideal value. F. SEQUENTIAL AND SIMULTANEOUS [Figure] [Figure]