S IMPLICIAL R EGULARIZATION
[Figure]
A BSTRACT
Inspired by the fuzzy topological representation of a dataset employed in UMAP (McInnes et al., 2018), we propose a regularization principle for supervised learning based on the preservation of the simplicial complex structure of the data.
We analyze the behavior of our proposal in contrast with the mixup (Zhang et al., 2018) framework on dimensionality reduction and classiﬁcation tasks. Our experiments show how simplicial regularization induces more appropriate learning biases and alleviates some of the shortcomings of state-of-the art methods for regularization based on data augmentation.
[Figure]
The richness of the class of neural networks as function approximators is well documented (Hornik, 1991; Cybenko, 1989; Telgarsky, 2015) and comes at the expense of having to regularize such functional families.
Recent works by Zhang et al.
(2018) and Verma et al.
(2019) have studied the properties of regularization induced by considering convex combinations of (latent representations of) data samples.
The authors argue that these techniques lead to learning “ﬂatter” representations.
The idea of using data augmentation as a form of regularization is not new.
Chapelle et al.
(2000) introduced the unifying framework of Vicinal Risk Minimization (VRM) in which the empirical risk objective R ERM ( f ) is modiﬁed via the introduction of an improved density estimate or vicinal distribution , P est ( x, y ) . The vicinal distribution acts as a probabilistic model from which artiﬁcial samples are generated towards regularizing the model f . [Equation]
The properties of the vicinal distribution P est induce different learning biases. For example, a Parzen windows estimate, corresponding to the addition of independent Gaussian noise to the covariates, is equivalent to ridge regularization in linear regression (Chapelle et al., 2000). In mixup , Zhang et al. (2018) propose a vicinal distribution via a data-agnostic augmentation procedure based on random convex combinations of arbitrary datapoints.
For example, for a multi-class classiﬁcation problem with inputs x i and one-hot targets y i , mixup generates virtual training examples (˜ x, ˜ y ) by sampling indices i and j independently and setting:
[Equation]
where λ follows a distribution with support on [0 , 1] , such as λ ∼ Beta ( α, α ) , for α > 0 .
In spite of their successful empirical results, the use of the mixup as a regularization scheme is mainly justiﬁed heuristically.
In this work, we study regularization based on the preservation of topological structures present in the dataset.
Our main contribution 1 is the introduction of simplicial regularization : we show how the preservation of the simplicial complex structure of the data naturally gives rise to a regularization framework that generalizes mixup .
Figure 1 illustrates the differences between sets of samples produced by the Parzen windows estimate, mixup , and our proposed simplicial regularization scheme.
S IMPLICIAL R EGULARIZATION
The core ingredient of mixup ’s vicinal distribution is the generation of random convex combinations of uniformly and independently selected datapoints form the dataset. Although this independent sampling contributes to the simplicity of the method, it is a na¨ıve assumption. There is a priori no reason that the addition of two arbitrary datapoints should carry semantic meaning, see the middle panel of Figure 1. Furthermore, even when the semantics of the input resulting from said addition can be determined, they may not coincide with any of the two original labels.
Simplicial regularization addresses both of these difﬁculties by giving up on the independence assumption.
Instead, we 1) infer a simplicial complex on the dataset following UMAP (McInnes et al., 2018), and 2) use the localized, correlated structure of this simplicial complex to design a vicinal distribution which better reﬂects the manifold-like properties of real data distributions. From now on, we concentrate on a a supervised learning task for which we have access to a dataset of input-target pairs D = { ( x i , y i ) } Ni =1 and a model class { f : X → Y | f ∈ F} , where X
and
Y are afﬁne spaces, and (cid:96) :
Y × Y → R is a loss function.
In the following we present the main ingredients of our proposed simplicial regularization framework. We refer the readers to Appendix A.1 for deﬁnitions of the concepts from algebraic geometry and fuzzy set theory employed below. I NFERRING THE SIMPLICIAL COMPLEX
UMAP is a dimensionality reduction technique, scalable to real world data and competitive with t-SNE (van der Maaten & Hinton, 2008). Given a dataset Φ = { φ i } Mi =1 embedded in a metric space, UMAP constructs a sparse , fuzzy simplicial complex K Φ = FuzzTop (Φ) . Informally, the information in K Φ reﬂects the ˜ n -nearest neighbors structure of Φ . Then UMAP proceeds to optimize the low-dimensional representations Ψ = { ψ i } Mi =1 such that the corresponding fuzzy simplicial complex K Ψ = FuzzTop (Ψ) matches K Φ as closely as possible.
In other words, we want the local neighborhood structure on the high and low-dimensional spaces to agree.
Appendix A.2 provides an overview of the inner workings of the FuzzyTop algorithm used to construct fuzzy topological representations of metric spaces. The ﬁrst step for applying simplicial regularization on the learning of the model f : X → Y is to construct the fuzzy simplicial complex representation of our training set K D = FuzzTop ( D ) 2 . Each of the simplices σ ∈ K corresponds to a subset of the training datapoints and is endowed with a value µ K D ( σ ) ∈ [0 , 1] denoting the membership of the simplex σ as an element of the set K D .
Furthermore, the membership of a simplex is upper bounded by the lowest membership of all its constituent lower-order simplices: for all σ (cid:48) ⊂ σ ∈ K D we have that µ K D ( σ ) ≤ µ K D ( σ (cid:48) ) . The simplicial complex formalism provides a natural generalization of mixup to higher-dimensional simplices. In fact, mixup can be interpreted as a complete graph which contains simplices of orderand 1 (the datapoints and all edges between them), with a constant membership function.
In contrast,
K D is carefully constructed so that the local neighborhood structure, and thus the generated linear interpolations, respect the manifold structure of the dataset. The sparsity level and the maximum simplex order of K D are determined by the neighborhood size ˜ n . This allows the method to scale to large datasets by avoiding the storage of a quadratic number of interactions among the datapoints. D ESIGNING THE VICINAL DISTRIBUTION
A morphism is a map between mathematical objects, which preserves the internal structure of the domain when transforming in to the codomain.
In set theory, morphisms are functions; in linear algebra, the vector space structure is preserved by linear transformations; while in topology, continuous functions preserve the neighborhood structure.
In this work, we advocate for a regularization paradigm that aims to preserve the inferred simplicial complex structure of the dataset, K D .
Consider two arbitrary simplicial complexes K ⊂ X and L ⊂ Y . The structure-preserving functions between simplicial complexes are called simplicial maps . A simplicial map f : K → L is a function which acts linearly on convex combinations over the simplices of K . Note that we do not require f to be a globally linear transformation, but only to commute with convex combinations of vertices forming a simplex in K .
In other words, if σ is a k -simplex of K with (ordered) vertex set σ 0 , we require that for every convex coefﬁcient k -vector λ :
[Equation]
The symbol f (cid:0) σ 0 (cid:1) denotes the collection of evaluations of the predictor f on the vertices of the simplex σ .
For example, for an M -class classiﬁcation problem, f (cid:0) σ 0 (cid:1) consists of a matrix of size dim ( σ ) × M , where row i represents the predicted label distribution for the i -th vertex of σ . Equation (3) highlights the fact that a simplicial map is {+uniquely determined by a base function f 0 : K 
 → L operating over the vertices of K , and its extension by convex interpolation on each simplex in K . When applying simplicial regularization to a supervised learning problem, the vertices of+}
K 0 D correspond to the training datapoints { x i } Ni =1 . Thus we take the base function as the training set labelling, i.e., f 0 ( x i ) = y i for x i ∈ K 0 D .
The learning problem amounts to ﬁt the model f to the extension of f 0 by convex interpolation over the simplices of K D . Enforcing Equation (3) in practice is far from trivial. However, we can use the loss function (cid:96) to measure “how far the function f is from being a simplicial map”. This suggests the deﬁnition of a vicinal risk which encourages f to preserve the simplicial structure of K D when mapping to L :
[Equation]
The parameter α > 0 inversely controls the level of interpolation which takes place within each simplex σ by imposing a symmetric Dirichlet distribution Dir ( dim ( σ ) , α ) over σ .
For small values of α , the distribution has peaks at the corners of the simplex recovering the standard ERM setting, while for α > 1 , the distribution becomes more skewed towards the center of the simplex. Equation (4) is presented in a suggestive notation to emphasize its connection to the VRM framework. In this case, the corresponding vicinal distribution is given by: [Equation]
Algorithm 1 describes the training pipeline for a supervised learning problem using simplicial regularization. Note how simplicial regularization is not embodied as an additive term alongside a supervised loss, but rather the regularized optimization objective is the vicinal risk itself. Appendix A.3 describes the FSC-Sample algorithm for efﬁcient sampling from a fuzzy simplicial complex.
Data: Dataset D = { ( x i , f 0 ( x i ) : = y i ) } Ni =1 , maximum iterations i max , learning rate η ,
optimizer O (e.g. SGD), Dirichlet interpolation schedule { α τ } i max τ =1 . 1 Compute fuzzy simplicial complex K D = FuzzTop ( D ) 2 Initialize predictor f θ : X → Y 3 for τ = 1 , . . , i max do
Sample σ ∼ K D (using FSC-Sample ( K D ) ), and λ ∼ Dir ( dim ( σ ) , α τ ) Compute Monte-Carlo estimator ˆ R SR = (cid:96) (cid:0) f (cid:0) λ (cid:62) σ 0 (cid:1) , λ (cid:62) f 0 (cid:0) σ 0 (cid:1)(cid:1) Perform optimizer step O on θ to minimize ˆ R SR with learning rate η 7 end
Remarks.
1) K D only needs to be computed once per dataset and can be stored and re-used for multiple training tasks. Although mixup does not require this computation, usually this cost is signiﬁcantly lower than that of the subsequent training loop.
It is possible to set up a schedule { α τ } for the Dirichlet interpolation coefﬁcient, such that α τ increases with τ during training.
This schedule encourages the approximation f θ to coincide with f 0 (the training labels) at the early stages of the optimization process, and then is progressively regularized in later iterations.
3) Unlike data augmentation methods relying on data invariances, such as translations or rotations for image data, our proposed augmentation framework is automatically inferred from geometry of the dataset .
E XPERIMENTS
In this section, we compare the performance of simplicial regularization and mixup on two tasks: i) multivariate regression on a synthetic dataset which has a simplicial complex structure and ii) multi-class digit classiﬁcation.
Dimensionality Reduction.
We generated a synthetic random point cloud X of size 100 in R 20 and ran UMAP on this dataset to produce a 2-dimensional embedding f 0 ( X ) = Y (see Figure 2), as well as the high-dimensional simplicial complex K X .
We then used several regularization methods to train neural networks of 2-hidden-layer neural and 100 units per layer to approximate the UMAPgenerated embedding X → Y .
That is, we want to ﬁnd a network f such that f ( x i )
≈ y i , where the discrepancy is measured by l = || · || 2 .
We compare between (unregularized) ERM, mixup , and simplicial regularization using the simplicial complex K X .
For the latter two, we used an increasing schedule for the regularization parameter α ∈ [10 − 4 , 1] during training.
[Figure]
As a sanity test, we are interested in comparing the performance of these methods when applied to data with structure satisfying the simplicial complex assumption.
We evaluate the generalization performance of these method according to the distributions P est (˜ x, ˜ y | K X , f 0 , α (cid:48) ) , using different interpolation strengths α (cid:48) .
Note that α (cid:48) measures the hardness of the task since preserving the simplicial complex structure becomes more important as α (cid:48) gets bigger and most of the evaluation points are sampled around the center of the simplices of K X . Moreover, α (cid:48) is independent of the schedule for α used during training. The values of log α (cid:48) accompany the markers of ERM and matching color across different methods represent the same value of α (cid:48) in evaluation
Figure 3 presents the mean and standard deviation of 30 rounds of evaluation of the generalization error of ERM, mixup and simplicial regularization.
As expected, the performance of unregularized ERM degrades quickly as the interpolation strength increases. Moreover, we observe that the models trained using simplicial regularization outpeform those trained using mixup when the evaluation task is highly dependent on the simplicial complex structure of the data.
In other words, when the unseen test data lies in the vicinity of local interpolations of the training data points, regularization methods agnostic to the topological structure of the data, like mixup , can lead to sub-optimal performance.
Digit Classiﬁcation.
In Figure 1 we illustrated the qualitative differences between the inputs produced by simplicial regularization, mixup and the Parzen windows estimate.
In this section, we consider a digit classiﬁcation task and expand our analysis of the perturbations on the data distribution induced by these techniques and the consequences at the input and prediction levels.
Figure 4 displays samples generated by interpolation of observed data points under the frameworks of simplicial regularization and mixup on the MNIST dataset (LeCun et al., 1998).
We argue that the inputs produced by simplicial regularization correspond to plausible, local variations of the data at the observed samples, in line with the tangent space approximation of the underlying data manifold.
This is in stark contrast with the samples produced by mixup . The un-relatedness of the samples being combined leads to inputs which do not correspond to likely variations of the data distribution, and thus lack semantic meaning.
Although it is possible to alleviate this issue by considering the use of mixup with low values of α , this approach leads to a vicinal distribution with low interpolation and thus, extremely close to the original data distribution, thus yielding un-regularized training.
The low α regime does not address the main issue of mixup : its use of structure-agnostic interpolations.
The mixup scheme is also problematic regarding its proposed labels for the generated interpolations.
The selection of random, unrelated, potentially very separated, data-points opens the possibility that some sections of the interpolation segment belong to regions of the input space which could be classiﬁed under a label which does not coincide with those of the endpoints .
The target proposed by mixup is oblivious to this observation and induces the classiﬁer to only produce predictions matching the classes of the sampled endpoints.
We trained two neural networks to classify MNIST digits using mixup and simplicial regularization with (weighted) cross-entropy loss.
We then sampled uniform, independent triplets of points (as in the mixup setting) and plotted the model predictions over the simplex comprising all convex interpolations on each triplet.
These results are presented in Figure 5.
Although both networks reached high predictive accuracy and their predictions coincide on the corners of the simplices, they exhibit vastly dissimilar behavior when evaluated at convex combinations of these corner points.
Note how the mixup framework leads to classiﬁers which (frequently) only produce predictions which agree with the endpoint classes, even though a human observer could assign a different label.
On the other hand, the local nature of the sampling in simplicial regularization allows for greater ﬂexibility in the predictions of the model, as the training objective never aims to prescribe behavior on distant regions based solely on local information.
A A PPENDIX A. D EFINITIONS AND N OTATION
For completeness, we provide deﬁnitions of several topological notions used in this manuscript.
While many of these deﬁnitions can be formulated in category-theoretical terms, we favour the purely geometric language for the sake of accessibility to the general machine learning community.
The deﬁnitions below follow closely the work of Munkres (1995).
σ in R n is the convex set spanned by k +1 linearly independent points { x 0 , . . .
, x k } ⊂ R n .
The points x i are called vertices , and the convex set spanned by any non-empty subset of these vertices is called a face of the K -simplex.
The set of l -simplices contained in σ is denoted σ l .
For example, the set of vertices of σ is denoted σ 0 and σ k only contains σ itself.
For k < l , σ l is empty.
The standard geometric K -simplex, denoted ∆ k , is the convex hull of the canonical basis of R k
+1 .
A simplicial complex K ⊂ R n is a collection of simplices in R n , possibly of varying dimensions, such that: (i) every face of a simplex of K is also in K , and (ii) the intersection of any two simplices of K is a face of each of them.
Intuitively, we can think of a simplicial complex K as made up of copies of standard simplices of several dimensions, glued together among some common faces.
We can organize the relevant information about a simplicial complex into the different skeleta K k , for k = 0 , 1 , . . .
, so that K k is the set of all K -simplices of K .
The structure-preserving functions between simplicial complexes are called simplicial maps .
Let
K and L be geometric simplicial complexes.
A simplicial map f : K → L is a function which acts linearly on convex combinations over the simplices of K .
Algebraically, consider an arbitrary m -simplex σ ⊂ K and a point x ∈ σ .
It is possible to represent x via its barycentric coordinates over σ as a weighted sum of the vertices σ 0 : there exists a probability m -vector t σ , such that x = (cid:80) mj
=1 t σj σ
j .
We say that f is a simplicial map if all x and for all σ containing x , we have that:
[Equation]
Note that a simplicial map f is uniquely determined by a function f 0 : K
→ L 0 operating over the vertices, and its extension by convex interpolation on each simplex in K .
Let α > 0 .
We denote by Dir ( k, α ) the symmetric Dirichlet distribution with density over the standard k − 1 simplex given by Γ( αk ) Γ( α )
k (cid:81)
ki
=1 x α − 1 i .
A fuzzy set is set S enriched with a membership function µ : S → [0 , 1] .
Given fuzzy sets ( S, µ ) and ( T, ν ) a morphism of between them is a function f : S → T such that for all s ∈ S , µ ( s ) ≤ ν ( f ( s )) .
Recalling the notion of a simplicial complex as a collection of simplices, we deﬁne a fuzzy simplicial complex as a simplicial complex K endowed with a membership function µ K : K → [0 , 1] , such that for all σ ⊂ σ (cid:48) , µ K ( σ ) ≥ µ K ( σ (cid:48) ) .
In other words, the membership of a simplex is upper bounded by the lowest membership of all its constituent lower-order simplices.
A strong negator is a monotonous decreasing, involutive function ¬ : [0 , 1] → [0 , 1] such that ¬ 0 = 1 and ¬ 1 = 0 .
For the remainder, we restrict our attention to the negator ¬ ( a )
= 1 − a .
A t-norm is a symmetric function (cid:62) :
[0 , 1] 2 → [0 , 1] satisfying for all a, b ∈ [0 , 1] :
• (cid:62) ( a, b ) ≤ (cid:62) ( c, d ) whenever a ≤ c and b ≤ d , • (cid:62) ( a, (cid:62) ( b, c )) =
(cid:62) ( (cid:62) ( a, b ) , c ) , and • (cid:62) (1 , a ) = a .
Given a t-norm (cid:62) , its complementary t-conorm under a negator ¬ is deﬁned by ⊥ ( a, b ) = ¬(cid:62) ( ¬ a, ¬ b ) .
A triplet ( (cid:62) , ⊥ , ¬ ) where (cid:62) is a t-norm, ⊥ is a t-conorm, and ¬ is a strong negator is called a De Morgan triplet if for all a, b ∈ [0 , 1] one has that ¬⊥ ( a, b )
= (cid:62) ( ¬ a, ¬ b ) .
The most common example of a De Morgan triplet is the one formed by (cid:62) prod ( a, b ) =
ab , ⊥ sum = a + b − ab and ¬ ( a ) = 1 − a .
Note how the t-norm and t-conorm express the probability of intersection and union of independent events.
Another important example arises by taking (cid:62) min ( a, b )
=
min( a, b ) , ⊥ max = max( a, b ) .
A De Morgan triplet ( (cid:62) , ⊥ , ¬ ) structure allows us to deﬁne operations between sets, or rather, between membership functions µ and ν on a common universal set U .
The complement of µ given by the composite function ¬ ◦ µ .
We deﬁne their union and intersection as the functions τ µ ∪ ν = ⊥ ◦ ( µ, ν ) and τ µ ∩ ν = (cid:62) ◦ ( µ, ν ) , respectively.
A. F UZZY REPRESENTATION OF A DATASET
The main building block of the UMAP dimensionality reduction technique is the construction of a carefully designed fuzzy topological representation of a given dataset X ⊂ R n .
In practice, this representation is stored as a weighted graph with restricted neighborhood size.
Next, we present a brief description of Algorithm 2.
The steps involved in this construction are illustrated in Fig.
For more details, please consult sections 2.2 and 3.1 of McInnes et al.
FinEPMet d i
We begin with a dataset X of N points embedded in R n and an arbitrary metric d .
For each data point i , we construct its set of ˜ n nearest neighbors N i and record the distance to its closest neighbor in ρ i =
min
j ∈N i d ( x i , x j ) .
We deﬁne a ﬁnite, extended, pseudo-metric space
d i around x i as: d i ( x i , x j ) = (cid:26) d ( x i , x j ) − ρ i , for j ∈ N i ∞ else.
We deﬁne µ i,j = exp (cid:18)
− d i ( x i , x j ) σ i (cid:19) for j ∈ N i , and select σ i > 0 in such a way that (cid:80) j ∈N i µ i,j = log ˜ n .
These ingredients allow us to deﬁne a fuzzy simplicial complex K i whose fuzzy 1-skeleton is given by the membership function µ i, ( · ) .
Note how all points in X outside of the neighborhood N i have membership zero.
Finally, this process is repeated for all data points in X and the individual simplicial complexes are joined via a pre-determined t-conorm ⊥ .
A. S IMPLICIAL COMPLEX EXTENSION AND SAMPLING
Algorithm 2 and its output are manageable for large-scale datasets due to the restricted neighborhood size and the use of a 1-skeleton representation.
Explicitly storing all simplices of higher dimensions quickly becomes a computational bottleneck due to the combinatorial number of possibilities.
In this section we present a construction that extends a fuzzy 1-skeleton into a fuzzy simplicial complex, and an efﬁcient algorithm to sample from it, while avoiding the explicit storage of memberships for high-dimensional simplices.
Algorithm
Data: 1-skeleton µ , neighborhoods N i , t-norm (cid:62) (e.g. (cid:62) min )
Result:
A random simplex sampled from the fuzzy simplicial complex extended from µ .
1 Sample u ∈ [1 , . . .
, N ] uniformly 2 N = N u ∪ (cid:4) 3 σ = [ u ] 4 while N (cid:54) = { (cid:4) } do foreach t ∈ N do ν t
= (cid:62) s ∈
σ µ s,t ; ν (cid:4) =
Sample u without replacement from N with probabilities proportional to ν if u (cid:48)
= (cid:4) then
Return σ else
Append u (cid:48) to σ end
N = ( N ∩ N u (cid:48) ) ∪ (cid:4) 14 end
Consider a (sparse) symmetric N × N matrix µ representing a fuzzy 1-skeleton and denote by N i the non-zero entries in row i .
It is easy to see that under a t-norm (cid:62) , the recursive membership:
[Equation]
is a valid extension of µ onto a fuzzy simplicial complex.
The base cases for these recursions are the 2 -simplex memberships µ K ([ i, j ]) =
µ i,j .
Trivially, this construction implies that a simplex σ has non-zero membership if all its constituent sub-simplices have non-zero membership.
Note that Algorithm 3 builds the required high-dimensional simplices on the ﬂy, and is guaranteed to terminate in at most ˜ n iterations of the outermost cycle.
The sampling of multiple simplices for the mini-batch setting can be easily parallelized.