O N THE N EED F OR T OPOLOGY -A WARE G ENERATIVE M ODELS FOR M ANIFOLD - BASED D EFENSES
A BSTRACT
Machine-learning (ML) algorithms or models, especially deep neural networks (DNNs), have shown signiﬁcant promise in several areas.
However, researchers have recently demonstrated that ML algorithms, especially DNNs, are vulnerable to adversarial examples (slightly perturbed samples that cause misclassiﬁcation).
The existence of adversarial examples has hindered the deployment of ML algorithms in safety-critical sectors, such as security.
Several defenses for adversarial examples exist in the literature.
One of the important classes of defenses are manifoldbased defenses, where a sample is “pulled back” into the data manifold before classifying.
These defenses rely on the assumption that data lie in a manifold of a lower dimension than the input space.
These defenses use a generative model to approximate the input distribution.
In this paper, we investigate the following question: do the generative models used in manifold-based defenses need to be topology-aware?
We suggest the answer is yes, and we provide theoretical and empirical evidence to support our claim.
I NTRODUCTION
Machine-learning (ML) algorithms, especially deep-neural networks (DNNs), have had resounding success in several domains.
However, adversarial examples have hindered their deployment in safety-critical domains, such as autonomous driving and malware detection.
Adversarial examples are constructed by an adversary adding a small perturbation to a data-point so that it is misclassiﬁed.
Several algorithms for constructing adversarial examples exist in the literature (Biggio et al., 2013;
Szegedy et al., 2013; Goodfellow et al., 2014b; Kurakin et al., 2016a; Carlini & Wagner, 2017; Madry et al., 2017; Papernot et al., 2017).
Numerous defenses for adversarial examples also have been explored (Kurakin et al., 2016b;
Guo et al., 2017; Sinha et al., 2017; Song et al., 2017; Tram`er et al., 2017; Xie et al., 2017; Dhillon et al., 2018; Raghunathan et al., 2018;
Cohen et al., 2019; Dubey et al., 2019).
In this paper, we focus on “manifold-based” defenses (Ilyas et al., 2017; Samangouei et al., 2018).
The general idea in these defenses is to “pull back” the data point into the data manifold before classiﬁcation.
These defenses leverage the fact that, in several domains, natural data lies in a low-dimensional manifold (henceforth referred to as the manifold assumptions) (Zhu & Goldberg, 2009).
The data distribution and hence actual manifold that the natural data lies in is usually unknown, so these defenses use a generative model to “approximate” the data distribution.
Generative models attempt to learn to generate data according to the underlying data distribution.
(The input to a generative model is usually random noise from a known distribution, such as Gaussian or uniform.)
There are various types of generative models in the literature, such as variational autoencoder (VAE) (Kingma & Welling, 2013), generative adversarial network (GAN) (Goodfellow et al., 2014a) and reversible generative models, e.g., real-valued non-volume preserving transform (Real NVP) (Dinh et al., 2016).
This paper addresses the following question:
Do manifold-based defenses need to be aware of the topology of the underlying data manifold?
In this paper, we suggest the answer to this question is yes .
We demonstrate that if the generative model does not capture the topology of the underlying manifold, it can adversely affect these defenses.
In these cases, the underlying generative model is being used as an approximation of the underlying manifold.
We believe this opens a rich avenue for future work on using topology-aware generative models for defense to adversarial examples.
Contributions and Roadmap.
We begin with a brief description of related work in Section 2.
Section 3 provides the requisite mathematical background.
Our main theoretical results are provided in Section 4.
Informally, our result says that if the generative model is not topology-aware, it can lead to a ”topological mismatch” between the distribution induced by the generative model and the actual distribution.
Section 5 describes our experimental veriﬁcation of our theoretical results and investigates their ramiﬁcations on a manifold-based defenses called Invert-and-Classify (INC) (Ilyas et al., 2017; Samangouei et al., 2018).
R ELATED WORK 2. G ENERATIVE MODELS
As a method for sampling high-dimensional data, generative models ﬁnd applications in various ﬁelds in applied math and engineering, e.g., image processing, reinforcement learning, etc.
Methods for learning data-generating distribution with neural networks include well-known examples of
Variational Autoencoders (VAEs) (Kingma & Welling, 2013) and variations of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014a; Radford et al., 2015; Zhao et al., 2016).
These generative models learn how to map latent variables into generated samples.
The VAE is a variational Bayesian approach, so it approximates a posterior distribution over latent vectors (given training samples) by a simpler variational distribution.
Similar to other variational Bayesian methods,
VAE tries to minimize the Kullback–Leibler divergence between the posterior distribution and the variational distribution by minimizing the reconstruction error of the autoencoder.
GANs represent another approach to learning how to transform latent vectors into samples.
Unlike other approaches, the GAN learns the target distribution by training two networks – generator and discriminator – simultaneously.
In addition to generating plausible samples, some generative models construct bijective relations between latent vector and generated samples, so that the probability density of the generated sample can be estimated.
Due to their bijective nature, such generative models are called to be reversible.
Some examples are normalizing ﬂow (Rezende & Mohamed, 2015), Masked Autoregressive Flow (MAF) (Papamakarios et al., 2017), Real NVP (Dinh et al., 2016), and Glow (Kingma & Dhariwal, 2018).
2. A PPLICATIONS OF GENERATIVE MODELS IN ADVERSARIAL MACHINE LEARNING
The DNN-based classiﬁer has been shown to be vulnerable to adversarial attacks (Szegedy et al., 2013; Goodfellow et al., 2014b; Moosavi-Dezfooli et al., 2016; Papernot et al., 2016;
Madry et al., 2017).
Several hypothesis try explaining such vulnerability (Szegedy et al., 2013; Goodfellow et al., 2014b; Tanay & Grifﬁn, 2016; Feinman et al., 2017), and one explanation is that the adversarial examples lie far away from the data manifold.
This idea leads to defenses making use of the geometry learned from the dataset – by projecting the input to the nearest point in the data manifold.
To learn a manifold from a given dataset, generative models can be exploited.
The main idea is to approximate the data-generating distribution with a generative model, to facilitate searching over data manifold by searching over the space of latent vectors.
The term Invert-and-Classfy (INC) was coined to describe this type of defense (Ilyas et al., 2017), and different types of generative models were tried to detect adversarial examples (Ilyas et al., 2017;
Song et al., 2017; Samangouei et al., 2018).
Usually, the projection is done by searching the latent vector that minimizes the geometric distance (Ilyas et al., 2017; Samangouei et al., 2018).
However, despite the promising theoretical background, all of those methods are still vulnerable (Athalye et al., 2018; Ilyas et al., 2017).
B ACKGROUND
We formally describe data generation, based on the well-known manifold assumption; data lies close to a manifold whose intrinsic dimension is much lower than that of the ambient space.
In our model of data generation, we provide a formal deﬁnition of data-generating manifold M on which the data-generating distribution lies such that M conforms to the manifold assumption.
3. R EQUIREMENTS
Real-world data tends to be noisy, so the data does not easily correspond to an underlying manifold.
We ﬁrst focus on an ideal case where data is generated solely from the manifold M without noise.
In the setting of classiﬁcation with l labels, we consider manifolds
M 1 ,...,M l ⊂ R n that correspond to the generation of data in each class i ∈ { 1 ,...,l } , respectively.
We assume those manifolds are pair-wise disjoint, i.e., M i ∩ M j = ∅ for any i /=
j .We set the data-generating manifold M as the disjoint union of those manifolds, M = ⊍ li
= 1 M i .
We assume M to be a compact Riemannian manifold with a volume measure dM induced by its Riemannian metric.
When a density function p M deﬁned on M satisﬁes some requirements, it is possible to compute probabilities over M via ∫ x ∈ M p M ( x ) dM ( x ) .
We call such M equipped with p M an dM as a data-generating manifold.
We refer to Appendix A and Appendix D.1 for details about deﬁnitions and requirements on p M .
In practice, data generation is affected by noise, so not all data lie on the data-generating manifold.
Therefore, we incorporate the noise as an artifact of data-generation and extend the density p M on M to the density p on the entire R n by assigning local noise densities on M .
We consider a procedure that (1) samples a point x o from M ﬁrst, and (2) adds a noise vector n to get an observed point ˆ x = x o + n .
Here, the noise n is a random vector sampled from a probability distribution, centered at x o , whose noise density function is ν x o , satisfying ν x ( n ) =
ν x ( ˆ x − x )
=
p M ( ˆ x ∣ x o = x ) .
3. E XTENDING DENSITY
When M is equipped with a density function p M and a measure dM that we can integrate over M , we can compute the density after random noise is added as follows.
[Equation]
Since ν x ( ˆ x − x ) is a function on x when ˆ x is ﬁxed, computing this integration can be viewed as the computing expectation of a real-valued function deﬁned on M .
Computing such expectation has been explored in Pennec (1999).
A demonstrative example is provided in Appendix B, and this extension is further discussed in Appendix D.2.
3. G ENERATIVE MODELS
A generative model tries to ﬁnd a statistical model for joint density p ( x ,y )
(Ng & Jordan, 2002).
We mainly discuss a speciﬁc type that learns a transform from one distribution D Z to another target distribution D X .
Commonly, a latent vector z ∼ D Z is sampled from a simpler distribution, e.g.,
Gaussian, then a pre-trained deterministic function G maps to a sample x = G ( z ) .
Speciﬁcally, we focus on reversible generative models to facilitate the comparison between the density of generated samples and the target density.
In this approach, the dimensions of latent vectors are set to be the same as those of the samples to be generated.
Also, for a given x , the density of x is estimated by the change of variable formula (equation (2) in Section 5.1).
3. I NVERT AND CLASSIFY (INC) APPROACH FOR ROBUST CLASSIFICATION
As the data-generating manifold M contains class-wise disjoint manifolds, there is a classiﬁer f on R n separating these manifolds.
If f separates the manifolds of M , any misclassiﬁed point should lie out of M .
Therefore, to change a correct classiﬁcation near a manifold, any adversary would pull a sample further out of the manifold.
By projecting misclassiﬁed points to the nearest manifold, we may expect the classiﬁcation to be corrected by the projection.
The INC method (Ilyas et al., 2017; Samangouei et al., 2018) implements this using a generative model.
The main idea of INC is to invert the perturbed sample by projecting to the nearest point on the data-generating manifold.
Ideally, the data-generating manifold M is accessible.
For any point ( ˆ x ,y ) with f ( ˆ x ) /=
y , out-of-manifold perturbation is reduced by projecting ˆ x to x ∗ on M .
The manifold M is unknown in practice.
However, as M is the data-generating manifold of D X , a generative model G for D X is trained to approximate M .
Then, searching over M is replaced by searching over latent vectors of G .
More details about INC implementations are described in Section 5.1.
T OPOLOGICAL PROPERTIES OF DATA FROM GENERATIVE MODELS
In this paper, we study the signiﬁcance of differences in the topological properties of the latent vector distribution and the target distribution in learning generative models.
Initial information about the topology of target distribution 1 is crucial to the generative model performance.
Speciﬁcally, if there is a difference in the number of connected components in the superlevel set between the target distribution and the distribution of the latent vector, then any continuous generative model G cannot approximate the target distribution properly (irrespective of the training method).
Due to the space limit, all proofs are presented in Appendix C.
4. T OPOLOGY OF DISTRIBUTIONS BASED ON SUPERLEVEL SETS
The data-generating manifold is a geometric shape that corresponds to the distribution.
However, this manifold is not accessible in most cases and we only have indirect access via the distribution extended from it.
Therefore, we consider ﬁnding a shape from the extended density so that this “shape” successfully approximates the data-generating manifold.
λ -density superlevel set.
We use the concept of λ -density superlevel set to capture geometric features of the density function.
Simply put, for a density function p and a threshold λ > 0 , the λ -density superlevel set L p,λ is the inverse image p − 1 [ λ, ∞] .
Our theoretical contribution is the conditional existence of a λ -density superlevel set reﬂecting the topology of the data-generating manifold under proper conditions on the noise density.
Assumptions on noise density.
For a family of densities { ν x } x ∈ M , we require the noise ν x to satisfy a number of assumptions.
These assumptions facilitate theoretical discussion about the superlevel set reﬂecting the data-generating manifold.
In the following deﬁnition, we denote a Euclidean ball of radius δ centered at x by B δ ( x ) .
Deﬁnition 1.
Let ν x be a family of noise densities.
• λ is small-enough if L ν x ,λ is nonempty for all x ∈ M , • λ -bounding radius δ x ,λ ∶= min { δ
∣ L ν x ,λ ⊆
B δ ( 0 )} is the smallest radius that B δ ( 0 ) contains L ν x ,λ .
When max x ∈ M δ x ,λ exists for some λ , we denote the maximum value as δ λ .
• λ -guaranteeing radius (cid:15) x ,λ ∶= max { (cid:15) ∣ B (cid:15) ( 0 ) ⊆
L ν x ,λ } is the largest radius that L ν x ,λ contains B (cid:15) ( 0 ) .
When min x ∈ M (cid:15) x ,λ exists for some λ , we denote the minimum value as (cid:15) λ .
Sufﬁcient conditions for the existence of these radii are discussed in Appendix D.3.
The properties of these radii are summarized in Lemma 1.
(The proof follows from Deﬁnition 1).
Lemma 1.
Let ν x be a family of noise densities and let λ be small-enough.
Then,
[Equation]
whenever δ λ and (cid:15) λ exist.
Figure 1 shows an example of superlevel set L x ,λ of noise ν x at a point x and its λ -bounding radius δ x ,λ and λ -guaranteeing radius (cid:15) x ,λ .
Finally, we deﬁne the continuous variation of noise densities ν x over changes of x ∈ M .
For the continuous variation, we require the continuity of both radii δ x ,λ and (cid:15) x ,λ as real-valued functions of x ∈ M for any ﬁxed value of λ .
Deﬁnition 2 (Continuously varying radii) .
Noise densities ν x have continuously varying radii if, for a ﬁxed small-enough λ , both λ -bounding radius δ x ,λ and λ -guaranteeing radius (cid:15) x ,λ are continuous functions of x ∈ M .
When noise densities have continuously varying radii, with the compactness of M , we can apply the extreme value theorem to guarantee the existence of both δ λ = max x ∈ M δ x ,λ and (cid:15) λ
= min x ∈ M (cid:15) x ,λ .
4. M AIN THEOREM
Our main theorem establishes, under the assumptions on noise densities from Section 4.1, the existence of a λ such that,
• (Inclusion)
The λ -density superlevel set L p,λ includes the data-generating manifold M .
• (Separation)
The λ -density superlevel set L p,λ consists of connected components such that each component contains at most one manifold M i .
Deﬁnition 3.
Consider a data-generating manifold M with density function p M .
For a radius (cid:15) > 0 , we deﬁne ω (cid:15) to be the minimum (over x ∈ M ) probability of sampling x ′ ∈ M in an (cid:15) -ball B (cid:15) ( x ) .
[Equation]
Deﬁnition 4 (Class-wise distance) .
Let ( X,d ) be a metric space and let M = ⊍ li
= 1 M i be a datagenerating manifold in X .
The class-wise distance d cw of M is deﬁned as,
[Equation]
With the deﬁnitions above, we proved the following main theorem.
Theorem 1.
Pick any small-enough threshold λ .
Fix a value λ ∗ ≤ ω (cid:15) λ and let δ ∗ =
δ λ ∗ be the λ ∗ - bounding radius.
If d cw of M is larger than 2 δ ∗ , then the superlevel set L p,λ ∗ satisﬁes the following properties
• L p,λ ∗ contains the data-generating manifold M .
• Each connected component of L p,λ ∗ contains at most one manifold M i of class i .
4. A PPLICATION TO THE GENERATIVE MODEL
We show an application of Theorem 1.
We denote the target distribution by D X , the latent distribution by D Z , and the distribution of G ( z ) where z ∼ D Z by D G ( Z ) .
Similarly, we denote the corresponding λ -density superlevel sets of densities by L X λ , L Z λ , and L G ( Z ) λ .
We assume the generative model G to be continuous.
Then, we get the following theorem regarding the difference between L Xλ and L G ( Z ) λ , in the number of connected components.
Theorem 2.
Let D Z be a mixture of n Z multivariate Gaussian distributions, and let the datagenerating manifold of D X contain n X components.
Let G be a continuous generative model for
D X using latent vectors from D Z .
Let λ
∗
be the threshold value from the Theorem 1.
If n Z < n X ,
L Xλ ∗ and L G ( Z ) λ ∗ do not agree on the number of connected components.
We can use this theorem to deduce the need for adequate information about the target distribution when training a generative model, especially if it is used for a security-critical application, e.g., INC.
Corollary 1.
If Theorem 2 is satisﬁed, there is a point ˆ x ∈ R n such that ˆ x /∈ L Xλ ∗
but ˆ x ∈ L G ( Z ) λ ∗ .
As a result, with density at least λ ∗ , G generates a point ˆ x that is unlikely to be generated by the target distribution.
Since INC is based on generations of G , the INC method can output an out-of-manifold point as a solution of optimization In Appendix D.4, Theorem 2 is generalized for more topological properties. spirals circles M 0 ∶ {( x 1 ,x 2 )∣ x
[Table]
= cos θ x 2 = sin θ }
M [Table] ∶ { [Table]
x 1 ,x
[Table]
[Table]
for θ ∈ [ 0 ,π ] [Table]  E XPERIMENTAL RESULTS In this section, we empirically demonstrate the consequence of the two theorems and explore their implication for the INC defense.
Our main goals are to provide (1) empirical support for the applicability of Theorem 2 and Corollary 1 via toy datasets, and (2) the improvement in INC performance using a class-aware generative model.
The main questions and the corresponding answers are shown below.
( Q1)
Can we experimentally verify the results of section 4.3?
Speciﬁcally, can we ﬁnd cases that the superlevel sets of D X and D G ( Z ) have different numbers of connected components?
( Q2)
How does INC fail when the generative model is ignorant of topology information?
( Q3)
Does the class-aware generative model improve the INC performance?
( A1) Theorem 2 and Corollary 1 can be veriﬁed by plotting the λ -density superlevel set.
Especially, we visualize the λ -density superlevel set of D G ( Z ) reﬂecting Theorem 2 and Corollary 1.
( A2) When generative model is not trained with topology information, naive INC may fail.
We found out two possible reasons regarding INC failure: (1) choice of a bad initial point and (2) out-of-manifold search due to non-separation of density superlevel set.
( A3)
The performance of INC is improved by training generative models with topology information on the target distribution.
We improved the average INC performance by decreasing the error induced by projection to 30% compared to the class-ignorant counterpart.
In the rest of this section, we provide a more detailed description of our experiments.
First, we brieﬂy describe the experimental setup in Section 5.1: datasets, latent vector distributions, training method, and INC implementation.
Then, Sections 5.2-5.4 describe the experimental results regarding the ﬁndings summarized above.
Section 5.5 contains an additional experiment illustrating the changes of decision boundaries by INC application.
5. E XPERIMENTAL SETUP
Datasets.
For all experiments, we use three toy datasets in R 2 : two-moons, spirals, and circles.
Table 1 summarizes the parameterizations 3 of each data-generating manifold and Figure 2 shows the plots of the corresponding data-generating manifolds.
To construct the training set, we ﬁrst sample 1000 points uniformly from each manifold M i , then each point is perturbed by isotropic
Gaussian noise N( 0 ,σ 2 I 2 ) with σ = 0 .
Before the training, each training set is standardized by a preprocessing of Scikit-learn package.
Latent vector distributions.
For latent vector distributions D Z , we prepared three different mixtures of n Z Gaussian distributions with n Z ∈ { 1 , 2 , 3 } .
When n Z = 1 , we simply use N( 0 ,I 2 ) .
When n Z = 2 , 3 , we arranged n Z Gaussian distributions along a circle of radius R = 2 . 5 , so that i -th
Gaussian has mean at µ i =
(− R sin ( 2 πin ) ,R cos ( 2 πin )) with σ = 0 .
n = 2 and σ = 0 .
3 for n = 3 .
Then, the uniform mixtures of the arranged Gaussian are used as D Z .
In Figure 3 (top row), we visualize the connected components corresponding to the latent vector distributions.
Training generative models.
Our experiments mostly use the Tensorﬂow Probability (Dillon et al., 2017) library that contains the implementation of reversible generative models.
Speciﬁcally, the Tensorﬂow Probability library contains an implementation of the Real NVP coupling layer that we used as a building block of our models.
The default template provided by Tensorﬂow Probability
[Figure] library was used to construct each Real NVP coupling layer with two hidden layers of 128 units.
Each model uses eight coupling layers that are followed by permutations exchanging two dimensions of R 2 except for the last coupling layer.
We describe the details of the training procedure of the generative models used in the experiments.
We prepared two different types of generative models: class-ignorant and class-aware.
The class-ignorant type is the usual Real NVP model.
This model uses the empirical estimation of negative log-likelihood over a training batch { x 1 ,..., x m } as its training loss.
[Equation]
The density p X of D X is estimated by applying the change of variables formula ,
[Equation]
where p Z is the density of D Z and ∂G ( z )
∂
z T is the Jacobian of G as a function from R n to itself.
The class-aware type is the Real NVP model trained with information about the number of connected components, i.e. the number of class labels l .
Using the number of labels, the densities p X and p Z can be decomposed as follows.
[Equation]
where p X,i ( x ) = p X ( x ∣ y
= i ) and each p Z,i is the i -th Gaussian component described above.
Since
Pr [ y = i ] is not generally known, the uniform distribution Pr [ y = i ]
= 1 l is used, where l is the number of classiﬁcation labels.
The main idea is class-wise training, i.e., training each p X,i from each p Z,i .
Applying the change of variables formula for each class i ,
[Equation]
Combining equations (3) and (4), we get the change of variables formula (2).
We deﬁne the class-wise loss function (cid:96) i for class-wise training as follows.
[Equation]
where m i is the number of training samples in class i .
Then, we train a generative model using the weighted sum of (cid:96) i as the training loss function.
[Equation]
Each model was trained for 30,000 iterations.
For each iteration, a batch of 200 random samples was chosen from two-moons and circles dataset, and a batch of 300 random samples was chosen from the spirals dataset.
For the choices of latent vector distribution, we chose the mixture of l − 1 Gaussians for the class-ignorant type, whereas we chose the mixture of l Gaussians for the class-aware type. [Figure]
V ISUAL VERIFICATION OF THEOREMS
The goal of this section is to verify Theorem 2 and the Corollary 1 by visualizing the superlevel set reﬂecting the statements.
Figure 3 shows the λ -density superlevel sets of densities of D G ( Z ) using the same threshold λ = 0 .
The ﬁrst row and the second row show the results from the class-ignorant version and those from the class-aware version, respectively.
Each column corresponds to each dataset.
All distributions are scaled for the standardization preprocessing before the training.
In general, superlevel set components are separated when the generative model is class-aware.
On the contrary, the class-ignorant generative models introduce connections between the components, as anticipated by Corollary 1.
Due to this connection, the class-ignorant generative models contain fewer connected components in their superlevel sets; this veriﬁes Theorem 2 for our choice of λ ∗ = 0 .
5. INC FAILURE DUE TO THE LACK OF INFORMATION ON THE DISTRIBUTION TOPOLOGY
We present how the non-separation of superlevel set components inﬂuences the performance of the
INC.
We provide two possible explanations of why the INC fails.
First, the bad initialization causes a suboptimal solution on a manifold not-the-nearest to the input.
Second, an artifact induced by the topological difference produces an out-of-manifold solution.
Figure 4 presents three visualized examples of INC with a class-ignorant generative model for twomoons.
In each plot, the black dot is the [Figure] [Figure] [Table]
[Figure] [Table]
Figure 4a is the INC optimization with an ideal start.
The initial point lies in the same manifold as the manifold closest to ˆ x .
Then, the INC optimization searches along the manifold, converging to a point close to ˆ x .
Figure 4b shows a case in which INC fails because of a bad initialization.
The initial point was chosen on a manifold not containing the desired solution, so the INC converged to a local optimum on the wrong manifold.
Our class-aware INC performs manifold-wise initialization to circumvent this issue.
Figure 4c shows that the INC failed due to an out-of-manifold search.
The INC converged in a wrong manifold, and a nontrivial amount of intermediate points were out of manifold, resulting in an out-of-manifold solution (see Figure 3d).
5. INC IMPROVEMENT VIA CLASS - AWARE GENERATIVE MODEL
We demonstrate that INC performance is improved by using class-aware generative models.
To measure the performance of the INC, 100 points are chosen uniformly from each manifold M i .
Then, each point x is perturbed by n x normal to the manifold at x , generating 200 adversarial points ˆ x = x ± r n x .
For all datasets, r = 0 .
2 is used for perturbation size.
We expect two types of INC to map ˆ x back to the original point x , as x is the optimal solution to (11).
We deﬁne the projection error of INC as ∥ INC ( ˆ x ) − x ∥ 2 , and collect the statistics of projection errors over all ˆ x .
Table 2 shows the projection error statistics for two types of INC.
Each pair of columns show the results on the indicated dataset.
For each pair, one column shows the error of the class-ignorant INC and the other column shows that of the class-aware counterpart.
Numbers in each cell are averages and standard deviations (in parenthesis) of the projection error.
For any dataset, the class-aware INC achieves lower projection errors.
Histograms of the projection errors are provided in Appendix E.
A DDITIONAL EXPERIMENTS FOR THE INC PERFORMANCE .
Finally, we present experiments to demonstrate the effect of the superlevel set discrepancy on the INC performance.
First, we begin with training support vector machines (SVMs) performing classiﬁcation tasks for our target distributions.
For training data, we randomly sampled 1000 training points from each data-generating manifold.
The baseline SVMs were intentionally ill-trained by using the high kernel coefﬁcient γ = 100 .
4 After training SVMs, we formed other classiﬁers by applying INC to ill-trained SVMs To explain, for each dataset, we have four types of classiﬁers as follows.
(1) Ill-trained SVM: Baseline classiﬁer (2) Ideal INC: Classiﬁer with INC using a direct access to the data-generating manifolds (3) Class-ignorant INC: Classiﬁer with INC using a topology-ignorant generative model (4) Class-aware INC: Classiﬁer with INC with using a topology-aware generative model
We want to emphasize that direct access to the data-generating manifold is not possible in general.
However, applying INC using direct access gives us an INC purely based on the geometry, so it is an ideal form of INC that should be approximated.
Also, since the class-ignorant INC is affected by a bad choice of an initial point, we reduced the effect of bad initialization by sampling more initial points and taking the best solution among the projection results.
For this number of initial choices, we [Figure] chose as many initial points as the number of manifolds, which was exactly the same as the number of initial points for the topology-aware INC model.
To demonstrate the improvement in the robustness of the model, we visualize the effect by depicting the decision boundary of each classiﬁer.
Speciﬁcally, we form a 300 × 300 grid on the domain of [− 3 , 3 ]×[− 3 , 3 ] and compute the classiﬁcation result.
The depicted decision boundaries are presented in Figure 5.
Each row corresponds to each dataset: two moons, spirals, and circles, respectively.
Each column corresponds to classiﬁers 1-4 described above, from the ﬁrst column to the fourth column, respectively.
From Figure 5, it is visually evident that the class-aware INC models provide more proper approximations to the ideal INC model compared to the class-ignorant INC models.
C ONCLUSION
We theoretically and experimentally discussed the necessity of topology awareness in the training of generative models, especially in security-critical applications.
A continuous generative model is sensitive to the topological mismatch between the latent vector distribution and the target distribution.
Such mismatch leads to potential problems with manifold-based adversarial defenses utilizing generative models such as INC.
We described two cases in which the INC failed: the bad initialization and the artifacts from the topological difference.
We experimentally veriﬁed that topology-aware training effectively prevented these problems, thereby improving the effectiveness of generative models in manifold-based defense.
After topology-aware training of generative models, the INC projection errors represented 30% of the errors of the topology-ignorant INC.
A CKNOWLEDGEMENT
Dr. Susmit Jha and Uyeong Jang’s internship at SRI International were supported in part by U.S. National Science Foundation (NSF) grants #1740079, #1750009, U.S. Army Research Laboratory Cooperative Research Agreement W911NF-17-2-0196, and DARPA Assured Autonomy under contract FA8750-19-C-0089.
The views, opinions and/or ﬁndings expressed are those of the author(s) and should not be interpreted as representing the ofﬁcial views or policies of the Department of Defense or the U.S. Government. This work is partially supported by Air Force Grant FA9550-18-10166, the National Science Foundation (NSF) Grants CCF-FMitF-1836978, SaTC-Frontiersand CCF-1652140 and ARO grant number W911NF-17-1-0405. R
A M ATHEMATICAL BACKGROUND A. G ENERAL TOPOLOGY
We introduce deﬁnitions and theorems related to general topology appeared in the paper.
For more details, all the deﬁnitions and theorems can be found in Munkres (2000).
Deﬁnitions in general topology.
We ﬁrst provide the precise deﬁnitions of the terms we brought from the general topology.
Deﬁnition 5 (Topological space) .
A topology on a set X is a collection T of subsets of X having the following properties.
1. ∅ and X are in T .
The union of the elements of any subcollection of T is in T .
The intersection of the elements of any ﬁnite subcollection of T is in T .
A set X for which a topology T has been speciﬁed is called a topological space .
For example, a collection of all open sets in R n is a topology, thus R n is a topological space.
If a topology can be constructed by taking arbitrary union and a ﬁnite number of intersections of a smaller collection B of subsets of X , we call B is a basis of the topology.
Pick a metric d in R n and consider B a set of all open balls in R n using the metric d .
The topology of R n can be constructed by taking B as a basis.
When this construction is possible, metric d is said to induce the topology .
Deﬁnition 6 (Metrizable space) .
If X is a topological space, X is said to be metrizable if there exists a metric d on the set X that induces the topology of X .
A metric space is a metrizable space X together with a speciﬁc metric d that gives the topology of X .
Since R n is equipped with Euclidean metric that induces its topology, R n is metrizable.
Continuity and the extreme value theorem.
Let X and Y be topological spaces. In the ﬁeld of general topology, a function f
∶ X → Y is said to be continuous , if for any subset V open in Y , its inverse image f − 1 ( V ) is open in X .
Moreover, if f is a continuous bijection whose inverse is also continuous, f is called a homeomorphism .
The notion of homeomorphism is important as it always preserves topological property, e.g., connectedness, compactness, etc., and this will be used in the further generalization of Theorem 2.
Here, we only introduce the generalized statement of extreme value theorem.
Theorem 3 (Extreme value theorem) .
Let f ∶ X
→ Y be continuous, where Y is an ordered set.
If
X is compact, then there exist points x and x in X such that f ( x ) ≤ f ( x ) ≤ f ( x ) for every x ∈ X .
Speciﬁcally, if a manifold M is a compact subset in R n , we may use X = M and Y = R .
Normal space and Urysohn’s lemma.
The Urysohn’s lemma was used to prove the Corollary 1.
We ﬁrst introduce the notion of normal space.
Deﬁnition 7 (Normal space) .
Let X be a topological space that one-point sets in X are closed.
Then,
X is normal if for each pair A , B of disjoint closed sets of X , there exist disjoint open sets containing
A and B , respectively.
Urysohn’s lemma is another equivalent condition for a space to be normal.
Theorem 4 (Urysohn’s lemma) .
Let X be a normal topological space; let A and B be disjoint closed subsets in X .
Let [ a,b ] be a closed interval in the real line.
Then there exists a continuous map f ∶ X —→ [ a,b ]
such that f ( x ) = a for every x in A , and f ( x ) =
b for every x in B .
To apply this lemma to R n , we only need the following theorem.
Theorem 5.
Every metrizable space is normal.
Since R n is metrizable, it is a normal space by Theorem 5.
Therefore, we can apply Urysohn’s lemma to any pair of disjoint subsets in R n , to show the existence of a continuous map f ∶ X → [ 0 , 1 ] .
A. D IFFERENTIAL GEOMETRY
We provide the deﬁnitions from differential geometry (Lee, 2003) used in the paper.
Manifold and tangent space.
Formally, topological manifold is deﬁned as follows.
Deﬁnition 8 (Manifold) .
Suppose M is a topological space.
We say M is a topological manifold of dimension k
if it has the following properties.
For any pair of distinct points x 1 , x 2 ∈ M , there are disjoint open subsets U 1 ,U 2 ⊂ M such that x 1 ∈ U and x 2 ∈ V . 2.
There exists a countable basis for the topology of M . 3.
Every point has a neighborhood U that is homeomorphic to an open subset ˜ U of R k .
There are different ways to deﬁne tangent space of k -dimensional manifold M .
Informally, it can be understood as geometric tangent space to M ⊂ R n at a point x ∈ M , which is a collection of pairs ( x , v ) where v is a vector tangentially passing through x .
Here we put a more formal deﬁnition of tangent space.
Consider a vector space C ∞ ( M ) , a set of smooth functions on M .
Deﬁnition 9 (Tangent space) .
Let x be a point of a smooth manifold M .
A linear map X ∶ C ∞ ( M ) → R is called a derivation at x if it satisﬁes
[Equation]
for all f,g ∈ C ∞ ( M ) .
The set of all derivations of C ∞ ( M ) at x forms a vector space called the tangent space to M at x , and is denoted by T x ( M ) .
Riemannian metric.
As tangent space T x
( M ) is a vector space for each x ∈ M , we can consider a inner product g bfx deﬁned on T x ( M ) .
Deﬁnition 10 (Riemannian metric) .
A Riemannian metric g on a smooth manifold M is a smooth collection of inner products g x deﬁned for each T x ( M ) .
The condition for smoothness of g is that, for any smooth vector ﬁelds X , Y on M, the mapping x ↦ g x (X∣ x , Y∣ x ) .
A manifold M equipped with a Riemannian metric g is called a Riemannian manifold.
B E XAMPLES
Computing density p M over a Riemannian manifold M .
This section presents example compu- tations of the probability computations from Section D.1 and Section 3.2 As a concrete example of computing density over a manifold, we use the following simple manifolds, so called two-moons in
R 2 .
[Equation]
We take M = M 0 ∪ M 1 as our example manifold.
Figure 6a shows the manifold of two-moons dataset plotted in different colors: M 0 in red and M 1 in blue.
First recall the following equation (equation (8) from the Section D.1).
[Equation]
where [ g X ( u ) ] is the k × k matrix representation of the inner product g X ( u ) at X ( u ) ∈ M .
Especially, when a manifold in R n is of dimension 1, i.e., parameterized curve γ ∶ [ a,b ] → R n , the integration (8) can be written in simpler way.
[Equation]
(5) where γ ′ ( t ) is the n -dimensional velocity vector at t ∈ [ a,b ] .
[Figure]
Let p M be a probability density function deﬁned on M .
As M is composed of two disjoint manifolds M 0 and M 1 , we consider conditional densities p 0 ,p 1 as follows.
[Equation]
Here, p M ∣ M 0 and p M ∣ M 1 represent the density function p M with its domain restricted to M 0 and M 1 , respectively.
By our deﬁnition of data-generating manifolds, Pr [ x ∈ M i ] corresponds to the probability of data generation for class i , i.e. Pr [ y = i ] .
For a concrete example of such density, uniform density for each manifold M i can be deﬁned as p i ( x ) = 1 π for all x ∈ M i .
Note that each manifold has parameterized curves in R 2 ,
[Equation]
with constant speed ∥ γ ′
( θ )∥ =
∥ γ ′ 1 ( θ )∥
= 1 at all θ ∈ [ 0 ,π ] .
Therefore, from equation (5),
[Equation]
For any measurable subset A ⊆ M , the probability for an event that x is in A can be computed as follows.
[Equation]
We can brieﬂy check all the requirements ( R1) , ( R2) , and ( R3) .
The computation of Pr [ x ∈ A ] is based on ( R1) , so ( R1) is satisﬁed trivially.
Also, p M is a function deﬁned only on M , thus ( R2) is clear, i.e. supp ( p M )
= { x ∈ R n ∣ p ( x ) >
} ⊆ M .
To check ( R3) , when A = M i , computing this integration will result in the exact probability Pr [ x ∈ M i ] =
Pr [ y = i ] , so when A = M , computing the integration will result in Pr [ y = 0 ]
+ Pr [ y = 1 ] = 1 , as desired in the requirements.
Extending density to R n .
We extend the domain to R n for the example of two-moon.
In Section 3, we deﬁned the noise density function to satisfy the following requirement.
( R0)
The translated noise density function , ν x ( ˆ x − x ) , is the density of noise n =
ˆ x − x being chosen for a given x .
Given x o = x , since adding noise n is the only way to generate ˆ x by perturbing x 0 , p ( ˆ x ∣ x o = x ) is equal to ν x ( n ) .
Under a proper noise density function, We show an example construction of the density extended from M satisfying the requirement ( R0) .
For simplicity, we choose isotropic Gaussian distribution,
N( 0 ,σ 2 I ) with the standard deviation σ for each dimension as the noise density function ν x for all x ∈ M .
Such noise density ν x deﬁned in R n can be written as follows.
[Equation]
By putting n x = ˆ x − x to density equation above,
[Equation]
Speciﬁcally, We assume an isotropic Gaussian distribution with σ = 0 .
as the noise density ν x for all x ∈ M .
By the equation (1), we have the following computation of density on ˆ x .
[Equation]
We can also check that the requirement ( R0) is satisﬁed by the construction; our construction (equation (1)) is based on ( R0) .
The computed density is shown in Figure 6b.
C P ROOFS
In this section, we provide the proofs for statements that appeared in Section 4.
C. P ROOF OF T HEOREM
To begin with, pick a value λ
such that the λ -density superlevel set
L ν x ,λ is nonempty for all x ∈ M .
As we use noise densities ν x described in Section 4.1, it is safe to assume that both λ -bounding radius
δ λ =
max x ∈ M δ x ,λ and λ -guaranteeing radius (cid:15)
λ
= min x ∈ M (cid:15) x ,λ exist.
Then, we can prove that, with a proper choice of threshold λ , the λ -density superlevel set includes the data-generating manifold.
Lemma 2.
Assume that noise densities have radii in Deﬁnition 1 for all x ∈ M and a small enough λ > 0 .
Then, for any x ∈ M , the density p ( x ) is at least ω (cid:15) λ , i.e. p ( x ) ≥ ω (cid:15) λ , where (cid:15) =
(cid:15) λ .
Proof.
By Lemma 1,
[Equation]
Then, we can lower bound the density p M ( x ) as follows.
[Equation]
This lemma shows that the thresholding the extended density p with threshold λ ∗ ≤ ω (cid:15) λ guarantees the superlevel set to include the entire manifold M .
Corollary 2.
For any threshold λ ∗ ≤ ω (cid:15) λ , the corresponding λ ∗ -density superlevel set L p,λ ∗ of the extended density p includes the data-generating manifold M .
Similarly, we show that, with a proper choice of threshold λ , each connected component of λ -density superlevel set contains at most one manifold.
Lemma 3.
Assume a family of noise densities satisﬁes the assumptions of Section 4.1.
Let λ > 0 be a value
such that the λ -density superlevel set
L ν x ,λ is nonempty for any x ∈ M .
Also, let δ =
δ λ be the maximum λ -bounding radius over M .
Then, for any ˆ x /∈ N δ ( M ) , the extended density value is smaller than λ , i.e. p ( ˆ x ) < λ .
Proof.
By Lemma 1,
[Equation]
Then, we can upper bound the density p ( ˆ x ) as follows.
[Equation]
This lemma says that the λ -density superlevel set is included by the δ -neighborhood N δ ( M ) of the data-generating manifold M .
Now, we can deduce the following main result.
Theorem 1.
Pick any λ ∗ ≤ ω (cid:15) λ threshold value satisfying the Corollary 2.
If the class-wise distance of data-generating manifold is larger than 2 δ ∗ where δ ∗ =
δ λ ∗ (the λ ∗ -bounding radius), then the superlevel set L p,λ ∗ satisﬁes the followings.
• L p,λ ∗ contains the data-generating manifold M .
• Each connected component of L p,λ ∗ contains at most one manifold M i of class i .
Proof.
The ﬁrst property is a direct application of Corollary 2 for λ ∗ = ω (cid:15) λ .
For the second property, since the class-wise distance of M is larger than 2 δ ∗ , the δ ∗ -neighborhood of manifolds are pairwise disjoint, i.e. N δ ∗ ( M i )
∩ N δ ∗ ( M j )
= ∅ for each
i /= j .
Therefore, N δ ∗ ( M ) has exactly k connected components N i = N δ ∗ ( M i ) ’
s.
By Lemma 3, δ ∗ -neighborhood N δ ∗ ( M ) contains the superlevel set
L p,λ ∗ , thus each connected component of L p,λ ∗ is in exactly one of N i ’s. Since M is contained in L p,λ ∗ , each M i is contained in some connected component C of L p,λ ∗ which is in N i .
Then, for any j /=
i , M j /⊂ C ⊂ N i , since
M j is in N j which is disjoint to N i .
Therefore, if a connected component C contains a manifold M i , then it cannot contain any other manifold.
P S ECTION
Theorem 2.
Let D Z be a mixture of n Z multivariate Gaussian distributions, and let D X be the target distribution from a data-generating manifold with n X manifolds.
Let G be a continuous generative model for D X using latent vectors from D Z .
Assume the Theorem 1 is satisﬁed, and let λ ∗ be the threshold value from the Theorem 1.
If n Z < n X , L Xλ ∗ and L G ( Z ) λ ∗ do not agree on the number of connected components.
Proof.
Since L Xλ ∗ is the results of Theorem 1, the number of connected components of L Xλ ∗ is at least n X .
However, since D Z is a mixture of Gaussians, for any value of λ (including the special case λ = λ ∗ ),
L Zλ can never have more than n Z connected components.
Since G is continuous, it preserves the number of connected components, thus L G ( Z ) λ ∗ =
G ( L Zλ ∗ ) has at most n Z connected components.
As n Z < n X , L Xλ ∗ and L G ( Z ) λ ∗ can never agree on the number of connected components.
Corollary 1.
If Theorem 2 is satisﬁed, there is a point ˆ x ∈ R n such that ˆ x /∈ L Xλ ∗
but ˆ x ∈ L G ( Z ) λ ∗ .
Proof.
Since n Z < n X , there exists a connected components ˆ C of L G ( Z ) λ ∗ containing at least two connected components of S Xλ ∗ .
Without loss of generality, assume ˆ C contains exactly two connected components C and C ′ .
By deﬁnition, λ -superlevel set is a closed set, so C and C ′ are disjoint closed sets.
In Euclidean space R n , the Urysohn’s lemma tells us that for any disjoint pair of closed sets
A,A ′ in R n , there is a continuous function f such that f ∣ A ( x ) = 0 and f ∣ A ′
( x ) = 1 for any x ∈ R n .
Especially, when A = C and A ′ =
C ′ , there exists a continuous function f such that,
• f ( x ) =
for all x in C • f ( x ) = 1 for all x in C ′
Consider S =
f − 1 ( 12 ) which is a separating plane separating C and C ′ .
If ˆ C ∩ S = ∅ , then ˆ C ∩ S = f − 1 [ 0 , 12 ) and ˆ C ∩ S = f − 1 ( 12 , 1 ] will be two open set in subspace ˆ C , whose union is ˆ C .
This implies that ˆ C is disconnected, which is a contradiction.
Therefore, ˆ C ∩ S should be nonempty, and any point x in ˆ C ∩ S is not in L Xλ ∗ .
D F URTHER DISCUSSIONS D. C OMPUTING DENSITY OVER A DATA - GENERATING MANIFOLD
When M is a Riemannian manifold equipped with a Riemannian metric g , we can compute probabilities over M .
There are two essential components of probability computation: (a) a density function p M and (b) a measure dM over M .
We assume p M and dM to satisfy the followings.
( R1)
For any measurable subset A ⊂ M , i
e., Pr [ x ∈ A ] = ∫ x ∈ A p M ( x ) dM ( x ) .
( R2) p is zero everywhere out of M , i
e., supp ( p M ) = { x ∈ R n ∣ p M ( x ) > 0 } ⊆
M ( R3)
For any ( x ,y ) , x is sampled from M i if and only if y =
i , i e. Pr [ x ∈ M i ] =
Pr [ y = i ]
When equipped with such p M and dM , we call M as a data-generating manifold.
Probability over a Riemannian manifold.
We show how to compute a probability of x being generated from a Riemannian manifold M .
We assume a k -dimensional manifold M equipped with a Riemannian metric g , a family of inner products
g x on tangent spaces T x M .
In this case, g induces the volume measure dM for integration over M .
If M is parameterized by x = X ( u ) for u ∈ D ⊆ R k , the integration of a density function p M on M is as follows.
[Equation]
where [ g X ( u ) ] is the k × k matrix representation of the inner product g X ( u ) at X ( u ) ∈ M .
In Appendix B, a concrete example of this computation will be provided.
D. D ENSITY EXTENSION OF THE S ECTION 3.
This section introduces some remaining discussions regarding our data-generating process from a data-generating manifold.
Relation to kernel density estimation.
While this extension is computing the density of compound distribution, it can be interpreted as computing expectation over a family of locally deﬁned densities.
Such an expected value can be observed in previous approaches of density estimation.
For example, if ν x is isotropic Gaussian for each x , the above integration is equivalent to the kernel density estimation, with Gaussian kernel, over inﬁnitely many points on M .
Observed property of the extended density.
In Figure 6b in Appendix B, we can observe that the extended density achieved higher values near the data-generating manifold.
We formalize this observation to discuss its implication to the INC approach.
Let d ( ˆ x ,M ) to be the minimum distance from ˆ x to the manifold M .
( C1)
For any given ˆ x , let y ∗ be the class label whose conditional density p ( ˆ x ∣ y
= y ∗) dominates p ( ˆ x ∣ y = i ) for i /=
y ∗ ,
[Equation]
and let M y ∗ be the manifold corresponding to y ∗ .
( C2)
For y ∗ satisfying ( C1) , we choose y ∗ such that the distance of ˆ x from the manifold d ( ˆ x ,M y ∗ ) is the smallest.
If there are multiple y ∗ satisfying both of ( C1) and ( C2) , we expect the following property to be true for all of those y ∗ .
( P1) Consider the shortest line from ˆ x to the manifold M y ∗ .
As ˆ x goes closer to M y ∗ along this line, ˆ x should be more likely to be generated as the inﬂuence of noise decreases when moving away from the manifold.
Therefore, we expect our density p M to have the following property.
[Equation]
Actually, this provides another justiﬁcation of INC.
In reality, the density conditioned by the label is not available even after running a generative model, so ﬁnding y ∗ with ( C1) is relatively hard.
If we only consider ( C2) without ﬁltering y ∗ via ( C1) , we are ﬁnding a point x ∈ M achieving the minimum distance to ˆ x , which is the optimization (11) above.
Then projecting ˆ x to the x ∗ , i.e. the solution of the optimization 11, can be explained by 10; when λ = 1 , p is the highest along the shortest line between ˆ x and x ∗ .
D. S UFFICIENT CONDITIONS FOR THE EXISTENCE OF RADII
We discuss the sufﬁcient conditions guaranteeing the existence of radii introduced in Deﬁnition 1.
Those sufﬁcient conditions are derived from natural intuition about the properties of distributions in most machine-learning contexts.
The ﬁrst intuition is that the inﬂuence of noise should diminish as observed sample ˆ x moves away from a source point x o .
Therefore, we formalize the noise whose density decreases as the noise n =
ˆ x − x o gets bigger.
We formalize boundedness of noise densities via the boundedness of their λ -density superlevel sets and continuity of noise density via the continuity of individual ν x .
Deﬁnition 11 (Center-peaked noise density) .
Noise density functions ν x are center-peaked , if for any source point x ∈ M and any noise vector n ∈ R n with ∥ n ∥ > 0 , ν x ( n ) <
ν x ( λ n ) for all λ ∈ [ 0 , 1 ) .
Deﬁnition 12 (Bounded noise density) .
Noise density functions ν x are bounded , if a λ -density superlevel set is nonempty, there is a radius δ by which the λ -density superlevel set is bounded, i.e.,
L ν x ,λ ⊆
B δ ( 0 ) where B δ ( 0 ) is the closed ball of radius δ centered at 0 .
Deﬁnition 13 (Continuous noise density) .
Noise density functions ν x are continuous , if ν x is continuous in R n , for any x ∈ M .
Under the conditions above, the radii in Deﬁnition 1 always exist.
Proposition 1.
If noise densities ν x are center-peaked, bounded, and continuous, any nonempty λ -density superlevel set L ν x ,λ has both λ -bounding radius δ x ,λ and λ -guaranteeing radius (cid:15) x ,λ .
Proof.
Let ν x be a center peaked, superlevel set bounded family of continuous noise densities.
Since ν x is continuous, superlevel set L ν x ,λ = ν − 1 x [ λ, ∞) is closed as an inverse image of ν x .
Therefore, its boundary ∂L ν x ,λ is contained in L ν x ,λ .
Because ν x is superlevel set bounded, superlevel set L ν x ,λ is bounded by a closed ball B δ ( 0 ) with radius δ ≥ 0 .
Since ν x is center peaked, a nonempty superlevel set L ν x ,λ always contains 0 as the maximum is achieved at 0 .
Moreover, there exists a closed neighborhood ball B (cid:15) ( 0 ) with radius (cid:15) ≥ 0 contained in the superlevel set L ν x ,λ .
Now it is enough to show that the minimum of δ and the maximum of (cid:15) exist.
Since L ν x ,λ is bounded, its boundary ∂L ν x ,λ is also bounded.
∂L ν x ,λ is closed and bounded, thus it is a compact set.
Therefore, the Euclidean norm, as a continuous function, should achieve the maximum r and the minimum r on ∂L ν x ,λ by the extreme value theorem.
From the choice of δ and (cid:15) , we can get, [Equation]
Therefore, we can ﬁnd the minimum δ x ,λ = r
and the maximum (cid:15) x ,λ = r .
D. G ENERALIZATION OF THE T HEOREM
We try generalizing the Theorem 2 to handle more concepts in topology.
Theorem 2 mainly uses a fact that the number of connected components of λ -density superlevel set is preserved by a continuous generative model G .
In algebraic topology, each connected component corresponds to a generator of 0 -th homology group
H 0 , and continuity of a function is enough to preserve each component.
In general, generators of i -th homology group H i for i > 0 are not preserved by a continuous map, so we need to restrict G further.
By requiring G to be a homeomorphism, we can safely guarantee that all topological properties are preserved by G ; therefore, we can generalize the Theorem 2 with a homeomorphic generative model
G .
To generalize the proof of the Theorem 2, we ﬁrst provide the sketch of the proof.
(1) λ ∗ -density superlevel set L Zλ ∗ of a mixture of n Z Gaussian distributions has at most n Z connected components.
(2) Since G is continuous, the number of connected components of L Gλ ∗ ( Z ) =
G ( L Zλ ∗ ) is same to the number of connected components of L Zλ ∗ , so it is also at most n Z .
(3) We choose λ ∗ so that L Xλ ∗ is included in δ ∗ -neighborhood of M .
(4) By assumption on the class-wise distance of M , δ ∗ -neighborhood of M has exactly same number of connected components to M ,
i
e., n X .
Therefore L Xλ ∗ has at least n X connected components.
(5) By (2) and (4) , we conclude that L Gλ ∗ ( Z ) and L Xλ ∗ do not agree on the number of connected components as long as n Z < n X .
In this proof, n Z corresponds to the maximal 0 -th
Betti number of L Zλ ∗ , i.e. the number of generators of H
( L Zλ ∗ ) .
If we keep using a mixture of Gaussians as latent vector distribution, all components of
L Zλ ∗ are contractible, so we may use 0 as the maximal i -th Betti number.
Also, n X corresponds to the 0 -th
Betti number of M and it worked as the minimal 0 -th
Betti number of L Xλ ∗ .
The condition on the class-wise distance of M is used to ensure n X to be a lower bound.
Combining these observations, we can get the following generalized statement.
Theorem 3.
Let D Z be a mixture of multivariate Gaussian distributions, and let D X be the target distribution from data-generating manifold M .
Let n i be the
i -th
Betti number of M .
Consider a generative model G is used to approximate D X using the latent vectors sampled from
D Z .
Assume that G is a homeomorphism from R n to itself.
Assume that data-generating manifold satisﬁes the conditions of the Theorem 1, and let λ ∗ be the threshold value that L Xλ ∗ corresponds to that superlevel set.
Assume that for some j > 0 , the homomorphism ι ∗ induced by the inclusion ι ∶ M → N δ ∗ ( M ) is injective.
If 0 < n j , L Xλ ∗ and L G ( Z ) λ ∗ do not agree on the number of connected component.
Proof.
Since L Xλ ∗ is the results of Theorem 1, it includes M and is included by δ ∗ -neighborhood
N δ ∗ ( M ) of M .
Deﬁne inclusions ι 1 ,ι 2 as,
• ι 1 ∶ M
→ L Xλ ∗ • ι 2 ∶ L Xλ ∗ → N δ ∗ ( M )
Clearly, ι = ι 2 ○ ι
[Equation]
Let ι ∗ 1 and ι ∗ 2 be induced homomorphisms of ι 1 and ι 2 , resp.
By the assumption, any generator [ a ] in H j ( M ) is injectively mapped to a nonzero generator ι ∗ ([ a ]) in H j ( N δ ∗ ( M )) .
Therefore, the j -th Betti number of N δ ∗ ( M ) is equal to that of M , i.e. n j .
Note that j -th Betti number is the rank of j -th homology group rank ( H j ( N δ ∗ ( M )))
Because ι ∗ 2 is a homomorphism from H j
( L X λ ∗ ) to H j ( N δ ∗ ( M )) , rank ( L Xλ ∗ ) ≥ rank ( H j ( N δ ∗ ( M ))) .
Therefore the j -th Betti number of L Xλ ∗ is at least n j .
However, since D Z is a mixture of Gaussians, for any value of λ (including the special case λ = λ ∗ ),
L Zλ does not have any generator of j -th homology group, so it has j -th Betti number 0 for all j > 0 .
Since G is homeomorphic, it preserves all the Betti numbers, thus L G ( Z ) λ ∗ =
G ( L Zλ ∗ ) has the same j -th Betti number.
As 0 < n j , L Xλ ∗ and L G ( Z ) λ ∗ can never agree on the number of connected components.
In Section 5.2, we see the Figure 3i from the circles dataset, which is a remarkable example that L Gλ ( Z ) has the same number of connected components, but does not have any loop (non-contractible circle).
This is empirical evidence of Theorem 3, so it is explained by mismatches in the topology of distributions.
Each concentric circle has Z as its ﬁrst homology group as circle contains exactly one generator.
However, latent vector distribution always has a trivial ﬁrst homology group, as any superlevel set of a mixture of Gaussians is a set of contractible connected components.
D. D ETAILS OF INC IMPLEMENTATIONS IN THE S ECTION
INC implementation.
We start from introducing the optimization for the ideal INC projection when the data-generating manifold M is available.
[Equation]
where d is a metric deﬁned on the domain X .
If perfect classiﬁcation on M is assumed (model is well- trained on M ) and ˆ x is close enough to the manifold of correct label, classiﬁcation f ( x ∗ ) is likely to be correct, since x ∗ is likely to lie on the correct manifold.
Since the data-generating manifold M is unknown, the INC approach runs the following optimization with before the classiﬁcation.
[Equation]
where d is a metric deﬁned on the domain X .
When INC is implemented with a reversible generative model G , for any given ˆ x ∈ R n there exists a trivial solution z ∗ =
G − 1 ( ˆ x ) to the optimization (12), achieving d ( G ( z ∗ ) , x ) = 0 .
This is even true for ˆ x out of the manifold, resulting in the situation that the output x ∗ =
G ( z ∗ )
=
ˆ x is still out of the data-generating manifold.
To manage this problem, we add another term penalizing a low density of latent vector to the objective function.
Thus, in our INC implementation, we solve the following optimization problem.
[Equation]
where α is the regularization factor and M is the maximum possible value of the density p
Z of the latent vector distribution.
For the choice of regularization factor, we used the same value α = 1 during the entire experiment.
To solve each optimization problem, we used the built-in adam optimizer (Kingma & Ba, 2014) in
Tensorﬂow package. For optimization parameters, we ran 100 iterations of adam optimizer using learning rate 0.01 with random sampling of z .
When implementing INC using a class-aware generative model, we used the following strategy to improve its robustness
• As the class-aware generative model generates each manifold from each Gaussian component, we ﬁrst sample initial points from each manifold by randomly choosing latent vectors z 1 ,..., z l from each Gaussian component.
• We run INC for i -th manifold by solving the following optimization.
[Equation]
where M i is the maximum value of i -th Gaussian component.
The regularization term is designed to penalize z which is unlikely to be generated by i -th Gaussian component, so we
only search in the range of i -th Gaussian component, i.e., i -th manifold.
• We choose the ﬁnal solution x ∗ i achieving the minimum d ( x ∗ i , ˆ x ) , breaking ties randomly.
Since each search is performed only on each submanifold, the artifact observed in Section 5.3 never appears during the optimization process.
Also, choosing initial points from each manifold prevents the initialization problem mentioned in Section 5.3.
D. D ISCUSSION ABOUT THE LIMITATION OF TOPOLOGICAL INFORMATION
Given a sufﬁcient number of connected components in the latent vector distribution, does the classaware training suggested in this paper result in a generative model that achieves manifold separation?
For this question, the answer is no , and the manifold separation depends on other factors, e.g., alignment of latent vector distribution, choice of training parameter, etc.
(a) Superlevel set of D Z
[Figure]
Figure 7b shows the superlevel set of D G ( Z ) from a class-aware training to learn the two-moons dataset when latent vector distribution is a mixture of two Gaussian distributions aligned horizontally (Figure 7a).
It is clear that in this case, the generative model induced a connection artifact even when the class-aware training was used.
We explain this by interpreting reversible generative models as dynamical systems (Weinan, 2017;
Chen et al., 2018; Grathwohl et al., 2018; Zhang et al., 2018).
To elaborate, a reversible generative model can be viewed as a dynamical system moving the latent vector distribution to the target distribution continuously in time.
When two Gaussian mixtures are aligned vertically, a reversible generative model is likely to learn how to move the upper (and lower) Gaussian distribution toward the upper moon (and the lower moon, respectively), without being affected by the entanglement of two moons.
However, moving the left (and right) Gaussian distribution toward the left moon (and the right moon, respectively) continuously in time is required to avoid the entanglement of two moons during the transition.
This case alludes that information about the topological properties may not be enough to learn a generative model separating manifolds, because it does not provide an understanding of information about how data-generating manifolds are aligned.
E M ORE EXPERIMENTAL RESULTS
We present more experimental results about the INC performance comparing topology-aware generative model to its topology-ignorant counterpart.
Histogram for projection error distributions in 5.4.
Figure 8 presents the histogram of the projection errors distributed from 0 to the diameter of the distribution.
Each row corresponds to each dataset, whereas the ﬁrst column and the second column represent the results from the topologyignorant model and the topology-aware model, respectively.
All histograms are normalized so that the sum of values adds up to 1.
To explain, the y -axis of each histogram is the estimated probability that INC achieves the projection error on the x -axis.
Not only can we observe the improved mean of projection errors in the histograms, but we can also check the reduced standard deviation, i.e., we get more consistent projection errors near the mean.
[Figure]