M INIMALISTIC U NSUPERVISED L EARNING WITH THE S PARSE M ANIFOLD T RANSFORM
Anonymous authors
Paper under double-blind review
A BSTRACT
We describe a minimalistic and interpretable method for unsupervised learning, without resorting to data augmentation, hyperparameter tuning, or other engineering designs, that achieves performance close to the state-of-the-art (SOTA) SSL methods.
Our approach leverages the sparse manifold transform [21] which unifies sparse coding, manifold learning and slow feature analysis.
With a onelayer deterministic sparse manifold transform, one can achieve 99 .
3% KNN topaccuracy on MNIST, 81 . 1% KNN top-1 accuracy on CIFAR-10 and 53 . 2% on CIFAR-100.
With a simple gray-scale augmentation, the model gets 83 .
2% KNN top-1 accuracy on CIFAR-10 and 57% on CIFAR-100.
These results significantly close the gap between simplistic “white-box” methods and the SOTA methods.
Additionally, we provide visualization to explain how an unsupervised representation transform is formed.
The proposed method is closely connected to latentembedding self-supervised methods and can be treated as the simplest form of
VICReg.
Though there remains a small performance gap between our simple constructive model and SOTA methods, the evidence points to this as a promising direction for achieving a principled and “white-box” approach to unsupervised learning, which has the potential to improve the learning efficiency significantly.
I NTRODUCTION
In the past few years, there has been tremendous progress in the unsupervised representation learning community.
This trend promises unparalleled scalability for data-driven machine learning in the future.
However, while we are celebrating the empirical advancements, the question of what a learned representation is and how exactly it is formed in an unsupervised fashion is still unclear.
Furthermore, it remains unclear whether there exists a set of common principles underlying all these unsupervised representations.
Many investigators have realized the importance of improving our understanding and have made pioneering steps trying to simplify the SOTA methods [18; 19; 113; 22], to establish the connections to classical methods [69; 4], to unify different approaches [4; 39; 97; 62; 55], to visualize the representation [9; 115; 15], and to analyze the methods from a theoretical perspective [3; 45; 106; 4].
The hope is that such understanding will lead to a theory that enables us to build simple, fully explainable, “white-box” models [14; 13; 71] from data based on first principles.
Such a computational theory could also provide guidance in understanding the principles governing unsupervised learning in the human brain [83; 31; 32; 65].
In this work, we make a small step toward this goal by trying to build a minimalistic “white-box” unsupervised learning model without deep networks, projection heads, augmentation, or various other engineering designs.
By leveraging two classical unsupervised learning principles, sparsity [81; 82] and low-rank spectral embedding [89; 104], we build a two-layer model which achieves non-trivial benchmark results on several standard datasets.
In particular, we show that a two-layer model based on the sparse manifold transform [21], which shares the same objective as latentembedding SSL methods [5], achieves 99.3% KNN top-1 accuracy on MNIST, 81.1% KNN topaccuracy on CIFAR-10, and 53.2% on CIFAR-100 without any data augmentation.
With a simple grayscale augmentation, it achieves 83.2% KNN top-1 accuracy on CIFAR-10 and 57% KNN topaccuracy on CIFAR-100.
These results make a step to close the gap between a “white-box” model and the SOTA SSL models [18; 20; 5; 116].
Though the gap is still clear, we argue that further closing it can potentially lead to a deeper understanding of unsupervised representation learning, and this is a promising path towards a useful theory.
We first begin the technical discussion below by addressing three fundamental questions.
Then in
Section 2, we revisit the formulation of SMT with a more general perspective and discuss how it is can be used to solve various unsupervised learning problems.
In Section 3, we present the benchmark results on MNIST, CIFAR-10, and CIFAR-100, together with visualization and ablations for a better understanding.
Further related literature is discussed in Section 4.
Finally, we make additional discussion in Section 5.
The main contributions of this paper are: 1) The original SMT paper explains SMT only from a manifold learning perspective. We provide a novel and crucial interpretation of SMT from a probabilistic co-occurrence point of view. The original SMT paper may create a misleading message that time is necessary to establish this transformation.
However, this is not true.
In this paper, we explain in detail how different kinds of localities can be leveraged to establish the transform.
We provide the benchmark results, which support the theoretical proposal in SMT.
We provide a connection between SMT and VICReg (and other SSL methods).
Since SMT is built purely from neural and statistical principles, this leads to better understanding of self supervised learning models.
Three fundamental questions:
What is an unsupervised (self-supervised) representation?
Essentially, any non-identity transformation of the original signal can be called a representation.
However, it is the useful ones that are interesting.
One general goal in unsupervised representation learning is to find a function that transforms raw data into a new space such that “similar” things are placed closer to each other.
Meanwhile, the new space should not be a collapsed trivial one 1 , i.e., the important geometric or stochastic structure of the data must be preserved.
If this goal is achieved, then naturally “dissimilar” things would be placed far apart in the representation space.
Where does “similarity” come from?
“Similarity” comes from three classical ideas, which were proposed multiple times in different contexts: 1) temporal co-occurrence [111], 2) spatial cooccurrence[90; 28; 34; 84], and 3) local metric neighborhoods [89; 104] in the raw signal space.
These ideas overlap to a considerable extent when the underlying structure is geometric 2 , but they can also conceptually differ when the structure is stochastic.
In Figure 1, we illustrate the difference between a manifold structure and a stochastic co-occurrence structure.
Leveraging these localities, two unsupervised learning methods emerge from these ideas: manifold learning and co-occurrence statistics modeling.
Interestingly, many of these ideas reach a spectral decomposition formulation or a closely related matrix factorization formulation [111; 89; 104; 27; 54; 44; 86; 46; 21].
The philosophy of manifold learning is that only local neighborhoods in the raw signal space can be trusted, and the global geometry emerges by considering all the local neighborhoods together, i.e., “think globally, fit locally”[92].
In contrast, co-occurrence [25] statistics modeling bears a probabilistic view, which complements the manifold view as some structures cannot be modeled by continuous manifolds.
A prominent example comes from natural languages, where the raw data hardly comes from a smooth geometry.
In word embedding [77; 78; 86], “Seattle” and “Dallas” may reach a similar embedding though they do not co-occur the most frequently.
The underlying reason is that they share similar context patterns [68; 117].
The probabilistic and manifold points of view complement each other for understanding “similarity”.
Now we have a definition of similarity; the next step is to construct a non-trivial transform such that similar things are placed closer.
How do we establish the representation transform?
The parsimonious principles: sparsity and low rank.
The general idea is that we can use sparsity to handle locality and decomposition in the data space to build a support (imagine the pillars of a bridge).
And then, we can construct our representation transform with low-frequency functions, which assign similar values to similar points on the support.
This process is called the sparse manifold transform [21].
U NSUPERVISED L EARNING WITH THE S PARSE M ANIFOLD T RANSFORM
The sparse manifold transform.
The sparse manifold transform (SMT) was proposed to unify several fundamental unsupervised learning methods: sparse coding, manifold learning, and slow feature analysis.
The basic idea of SMT is: 1) first to lift the raw signal x into a high-dimensional assume space by a non-linear sparse feature function f , i.e. ⃗α = f ( ⃗x ) , where locality and decomposition are respected.
2) then to linearly embed the sparse feature ⃗α into a low-dimensional space, i.e., ⃗β
=
P ⃗α , such that the similarity or “trajectory linearity” is respected.
As we mentioned earlier, for each data point x i , we can choose to use either its local neighbors in the raw signal space, its spatial co-occurrence neighbors, or its temporal co-occurrence neighbors to define similarity.
Let’s denote the neighbors of x i as x n ( i ) , the optimization objective is the following:
[Equation]
where ⃗α = f ( · ) is a sparse feature extraction function, e.g. 1-sparse vector quantization f vq and sparse coding f sc .
Later we will provide several other examples.
Similarity can be considered as a first-order derivative (see Appendix L for detailed explaination).
A more restrictive requirement is linearity [89], which is the second-order derivative.
Then the objective is changed to a similar one:
[Equation]
where (cid:80) j ∈ n ( i ) w ( i, j )
= 1 .
Several ways to acquire w ( · , · ) are: to use locally linear interpolation, temporal linear interpolation, or spatial neighbor linear interpolation.
If we write the set { ⃗α i } as a matrix A , where each column of A corresponds to the sparse feature of a data point, then both
Optimization 1 and Optimization 2 can be formulated as the following general formulation [21]:
[Equation]
where D is a differential operator, V = 1 N AA T and N is the total number of data point.
D corresponds to a first-order derivative operator in Optimization 1 and a second-order derivative operator in Optimization 2.
For Optimization 1, D ji = − 1 for j ∈ n ( i ) , D ii = | n ( i ) | , and D ji = 0 otherwise. D has M columns, where M denote the number of pair of neighbors. D = [ d 1 , d 2 , · · · d M ] and for k th pair of neighbors x i , x j , we have d k i = 1 and d k j = − 1 .
All other elements of D are zero. For Optimization 2, D ji = − w ( i, j ) for j ∈ n ( i ) , D ii = 1 , and D ji = 0 otherwise.
The solution to this generalized eigen-decomposition problem is given [108] by P = UV − 12 , where U is a matrix of f trailing eigenvectors (i.e. eigenvectors with the smallest eigenvalues) of the matrix
Q = V − 1 2 ADD T A T V − 1 2 .
Please note that if U contains all of the eigenvectors of Q , then the solution of SMT reduces to the whitening matrix.
We can generalize f to an arbitrary function as long as it can capture the locality and structures in the signal space.
If we choose f as a quadratic expansion function, Optimization 1 would reduce to slow feature analysis [111].
Manifold point of view: a 2D nonlinear manifold disentanglement problem.
Let’s consider a spiral manifold disentanglement problem in a 2D space shown in Figure 2(a).
As it is not a linearlyseparable problem, to construct a representation function to disentangle the two spiral manifolds, we need to leverage nonlinearity and give the transform function enough flexibility to respect the locality in the data space.
Let’s first use 1-sparse sparse coding to learn an overcomplete dictionary 3 , i e. a set of K landmarks [98; 27], to tile the space.
This dictionary serves as the support of the representation function, as shown in Figure 2(b).
This dictionary turns a data point ⃗x into a 1-sparse vector ⃗α = f ( ⃗x ) =
argmin ⃗α ∥ ⃗x − Φ ⃗α ∥ , s.t.
∥ ⃗α ∥
= 1 , ∥ ⃗α ∥ 1 = 1 ; ⃗α ∈ IR K is an 1-hot vector and denotes the closest dictionary element ⃗ϕ to ⃗x .
Next, we seek a linear projection P to embed the sparse vector ⃗α to a lower dimensional space, ⃗β = Pf ( ⃗x ) and ⃗β ∈ IR d , which completes the transform.
Each column of P ∈ IR
d × K corresponds to a d -dimensional vector assigned to the corresponding entry of α .
The i th row, P i, · , of P is an embedding dimension, which should assign similar values to the nearby landmarks on the manifolds.
As we mentioned earlier, there are three localities to serve as similarities, where they behave similarly in this case.
With either of the three, one can define the Optimizationor Optimization 2.
In this simple case, the different similarity definitions are almost identical, leading to a spectral solution of P .
Let’s choose a simple construction [69]: given a data point ⃗x ∈ IR 2 , add Gaussian noise ϵ 1 , ϵ 2 to it to generate two augmented samples ⃗x 1 = x
+ ϵ 1 and ⃗x 2 = x 2 + ϵ 2 , which are marked as the red and green crosses in Figure 2(b) and are considered as similar data points.
f ( ⃗x 1 ) and f ( ⃗x 2 ) are marked as red and green dots (landmarks).
So the objective in Optimization 1 is to find P such that E x ∼ p ( x ) ,ϵ 1 ,ϵ 2 ∼N (0 ,I )
∥
Pf ( x 1 + ϵ 1 ) − Pf ( x 2 + ϵ 2 ) ∥ 2 is minimized.
We show the solution in Figure 2(c) with the first four embedding dimensions, { P 1 , · , P 2 , · , P 3 , · , P 4 , · } plotted, where the color reflect their values at each dictionary element.
As we can see, each embedding dimension, e.g. P i, · , is essentially a low-frequency function.
P 1 , · , P 2 , · are piecewise constant functions on the union of the manifolds.
Essentially, either P 1 , · or P 2 , · will suffice to determine which manifold a point belongs to.
P 2 , · , P 3 , · are non-constant linear functions along the manifolds.
A linear combination of P 3 , · and P 4 , · will preserve the coordinate information of the manifolds.
Now with both the sparse feature function f and the spectral embedding P , we have constructed a sparse manifold transform, i.e. ⃗β =
Pf ( ⃗x ) .
In Figure 2(d), we show how each dimension of this transform assigns different values to each point in the original 2D space, and the points from the same manifold are placed much closer in the representation space as shown in Figure 2(f).
Probabilistic point of view: co-occurrence modeling.
Many related variations of spectral and matrix factorization methods have been widely used to build continuous word representations.
SMT formulation bears a probabilistic point of view, which complements the manifold perspective.
A notable example is the word co-occurrence problem.
Here we present an SMT-word embedding.
Given a corpus and a token dictionary, we can treat each token as a 1-hot vector ⃗α .
Then we can seek a linear projection P such that tokens co-occur within a certain size context window are linearly projected closer.
After we solve Optimization 1, each column of P corresponds to a continuous embedding of an element from the token dictionary.
We train SMT on WikiText-103 [76] and provide the visualization result in Appendix A. As we discussed earlier, “Seattle” and “Dallas” may reach a similar embedding though they do not co-occur the most frequently.
The probabilistic view of SMT explains the phenomenon that two embedding vectors can be close in the representation space if they frequently appear in similar contexts, though they may not be close in the signal space.
Sparsity, locality, compositionality, and hierarchy.
(a) Sparsity: In the sparse manifold transform, sparsity mainly captures the locality and compositionality, which builds a support for the spectral embedding.
Spectral embedding, on the other hand, would assign similar values to similar points on the support.
(b) Locality:
In a low-dimensional space, e.g., the 2D manifold disentanglement case, with enough landmarks, using 1-sparse feature f vq to capture locality will suffice, and SMT can directly solve the problem as we showed earlier.
Please note that, in this simple case, this process is not different from manifold learning [89; 104] and spectral clustering [80].
One can also use a kernel, e.g. RBF kernel or the heat kernel, to respect locality by lifting to a higher or infinite dimensional space [95; 111; 6; 102; 8].
(c) Compositionality: If the signal comes from a highdimensional space, e.g. images, then we may need to build the representation for image patches first since even local neighborhoods can become less reliable in high-dimensional space with finite samples.
If the patch space is simple enough, e.g. MNIST, using 1-sparse vector quantization as the sparse feature f vq would be sufficient to capture the locality in patch space.
As we will show later in Section 3, the SMT with 1-sparse feature f vq achieves over 99.3% top-1 KNN accuracy.
For more complex natural images, e.g. CIFAR or ImageNet, compositionality exists even in the patch space.
So 1-sparse feature becomes less efficient and may compromise the fidelity of the representation.
As we will show, the general sparse feature will achieve better performance than the 1-sparse feature in this case.
(d) Hierarchy: For high-resolution images, we may need to stack several layers of transforms in order to handle the hierarchical representation [107; 35; 52].
-wise L2 normalization. We use tensor ⃗z as a vector for an image representation. For MNIST experiments, we choose the patch embedding dimension as 32 by default. For CIFAR experiments, we choose the patch embedding dimension as 384 by default.
SMT representation for MNIST and CIFAR natural images.
We use SMT to build representation for image patches and aggregate patch representation of an image as the image representation.
For an S × S image, we extract every T × T image patches as shown in Figure 3.
Each of the image patches is whitened and L2-normalized , which resembles the early visual system processing [109; 74].
Given an image, we denote the pre-processed image patches as { ⃗x ij } , where i indicates the vertical index of the patch and j indicates the horizontal index.
For each image patch { ⃗x ij } , we define its neighbors to be other patches within a certain pixel range from it on the same image, i.e., n ( ij ) = { lm ; | i − l | ≤ d and | j − m | ≤ d } .
Given a sparse feature function f , we would like to map these neighboring patches to be close in the representation space.
If we choose d = S − T + 1 , then all the patches from the same image are treated as neighbors to each other.
Given f , d , and a set of training images, we can calculate ADD T A T , V , and solve Optimization 1 for P 4 .
The representation of patch ⃗x ij is ⃗β ij =
Pf ( ⃗x ij ) with a L2 normalization.
If we denote all of the patch representations from an image as tensor B , the image representation is ⃗z =
L2 normalize pw ( avg pool ( B ; ks , stride )) .
avg pool ( · ; ks , stride ) denotes the average pooling operator, where ks and stride are the kernel size and stride size respectively.
L2 normalize pw denotes a point-wise L2 normalization.
We use tensor ⃗z as a vector for an image representation.
The detailed settings, e.g. d, ks , stride, etc., are left to the experiments section.
In this work, we primarily use the soft-KNN classifier to evaluate the quality of representation.
Connection to the deep latent-embedding methods.
There are important connection between the
SOTA latent-embedding SSL methods[18; 5].
In particular, Optimization 1 can be considered the simplest form of VICReg [5], whereas the VC regularization is replaced by a hard constraint, and a sparse feature function replaces the deep neural encoder.
See exact formulation in Appendix K. This connection is even stronger given the evidence that VICReg is essentially building a representation of image patches [22].
In this sense, SMT can be considered the simplest form of VICReg.
E MPIRICAL R ESULTS
In this section, we first provide the benchmark results on MNIST, CIFAR-10, and CIFAR-100.
Qualitative visualization is also provided for a better understanding.
We also compare 1-sparse feature f vq and a more general sparse feature f gq .
⃗α = f vq ( ⃗x ; D ) refers to the 1-sparse vector quantization sparse feature, where α i = 1 for i = argmax i ∥ ⃗x − D · ,i ∥ 2 and D is a dictionary.
We denote the SMT representation with f vq sparse feature as SMT-VQ.
⃗α = f gq ( ⃗x ; D, t ) refers to a more general sparse function, where each column of dictionary D is an image patch and t is a given threshold.
For ⃗α = f gq ( ⃗x ; D, t ) , the i th entry of ⃗α is 1 only if the cosine similarity between ⃗x and i th dictionary element pass the threshold, i.e., α i = 1 if cos ( ⃗x, D · ,i ) ≥ t and α i =
otherwise.
We denote the SMT representation with f gq feature as SMT-GQ.
We find f gq achieves similar performance as the sparse coding feature in benchmarks, and it’s more efficient in experiments since it does not involve a loopy inference.
For both MNIST and CIFAR, we use 6x image patches, i.e., T = 6 .
All the image patches are whitened, and the whitening transform is described in Appendix C.
3-Pixel ( d = 3 ) and Whole Image ( d = 28 − 6 + 1 ).
We use 32 as the embedding dimension.
As shown in the table, GloVe embedding works slightly worse than SMT embedding.
Co-Occurrence Context Range
[Equation] [Table]
MNIST.
As we mentioned earlier, once we decompose the MNIST image into patches, 1-sparse feature f vq is sufficient to build a good support of the patch space.
The dictionary D ∈ IR 36 × K is learned through 1-sparse sparse coding, which is equivalent to a clustering process with a GPU implementation. , which can be implemented with any clustering algorithm, the centroid of each cluster will serve as dictionary elements.
The detailed learning procedure is left to Appendix D.
The inference is equivalent to vector quantization.
In Table 1, we compare different dictionary sizes and different spatial co-occurrence range.
The dimension of the embeddings is 32 dimensional, where further increasing the dimension does not improve the performance.
Treating each image patch as a token and the entire image patches as sentences, we can also train a GloVe word embedding model on MNIST patches.
We provide a GloVe embedding with optimized settings.
As we can see in Table 1, GloVe embedding works slightly worse in this case, even with a large dictionary size.
Further, we can find that a small context range works slightly better for MNIST.
This is probably due to that the patch space of MNIST is closer to a manifold, and co-occurrence is less informative in this case.
To further verify this hypothesis, we provide an ablation study on embedding dimensions in Table 2.
Even if we reduce the embedding dimension to 4 5 , the top-1 KNN accuracy is still 98 .
This implies that the MNIST patch space might be close to a manifold like high-contrast natural image patches [67; 26; 12].
[Table]
CIFAR-10 and CIFAR-100. For the CIFAR benchmarks, we compare both f vq ( · ; D ) and f gq ( · ; D, t ) .
The dictionary D used in f vq is learned through 1-sparse sparse coding, and the dictionary D used in f gq is just randomly sampled from all image patches for efficiency purposes.
First, with gray-scale patches, we find that f vq and f gq work equally well.
However, since f vq with color patches underperforms that with gray-scale patches, we choose to experiment mostly with f gq on
CIFAR benchmarks.
For f gq , t is set to 0 .
3 with color patches and 0 .
45 with gray-scale patches.
For CIFAR experiments, we choose the patch embedding dimension as 384 by default.
Further, with color patches, dropping the first 16 embedding dimensions of P improves the accuracy by about 1%.
SMT training uses both the original training set and the horizontally flipped version.
On the other hand, SSL models use image flip augmentation as their default setting.
For the “Original + Gray Scale” augmentation, SMT-GQ simply concatenates the embedding of color patch embedding and the gray-scale patch embedding.
With a two-layer deterministic construction using only image patches, SMT significantly outperforms the SOTA methods, SimCLR and VICReg, with no color augmentation.
Please note that SimCLR and VICReg still use random-resized crop augmentation.
With grayscale-only color augmentation, SMT-GQ achieves similar performance as SimCLR and VICReg.
SMT-GQ with “+gray scale” augmentation performs on par with SimCLR and VICReg without colorjitter and several additional color augmentations.
SimCLR and VICReg, with full data augmentation, outperform SMT-GQ.
However, we did not find a straightforward way to add these color augmentations to SMT-GQ.
We can observe the same trend on CIFAR-100.
Interestingly, SMT-GQ always significantly outperforms SimCLR and VICReg before we add color jittering and additional data augmentations.
We provide further ablation study in Appendix E to show two components of sparse manifold transforms: sparse feature and spectral embedding are both crucial for uation result of SMT embedding and SSL pretrained models using KNN classifiers.
We evaluated various color augmentation schemes. For “original image”, we use only “horizontal flip” data augmentation for SMT, and use both “horizontal flip” and “random resized crop” data augmentation for SSL methods.
For “Full”, we added both “grayscale” and “colorjitter” color augmentation on top of other data augmentations in “original image”.
We tried two sparse feature for SMT: f vq and f gq .
For f gq , we tried two dictionary sizes: 8192 and 65536.
Both SSL models uses ResNet-18 as backbone and is optimized for 1000 epochs through propagation; SMT uses a 2-layer architecture and is analytically solved , which can be considered as 1 epoch.
[Table]
building good representations. Only if we use spectral embedding, will the performance scale with a larger dictionary. The performance of SMT will significantly degrade if we replace spectral embedding with some other dimension reduction algorithm that doesn’t respect locality. More ablation studies are included in the appendix, including ablation study on: context range (Appendix G), number of training samples (Appendix H), and evaluation method (Appendix I).
[Table]
Visualization.
For a better understanding
[Table]
we provide a visualization of the patch embedding space with the 1-sparse feature in Figure 4.
Given a dictionary D , each dictionary element is an image patch in the whitened space.
For a dictionary element shown in the red box, we can visualize its nearest neighbors, measured by cosine similarity, in the embedding space.
All the image patches are unwhitened before visualization, where the detail is left to the Appendix C.
In Figure 4, we can see that the nearest neighbors are all semantically similar.
a) is a slightly curved stroke of a digit in MNIST.
b) shows the patch of the center of a digit “4”.
c) shows a part of a “wheel” of a car in CIFAR.
d) shows a part of “animal face”.
These results are similar to the 2D results shown in
Figure 2.
The sparse feature respects the locality in the data space and provides a support for the representation transform and spectral embedding, assigning similar values to similar points on the support. Overall the representation transform brings similar points in the data space closer in the representation space.
Additional visualization for SMT on CIFAR-10 dataset is in Appendix M.
M ORE R ELATED W ORKS
There are several intertwined quests closely related to this work.
Here, we touch them briefly.
The state-of-the-art unsupervised methods.
Deep learning [63; 66] is the engine behind the recent revolution of AI.
Given the right craftsmanship, deep neural networks are powerful optimization tools to allow us express various engineering objectives conveniently.
It has also revolutionized the unsupervised visual representation learning in the past few years [112; 47; 18; 5].
Understanding the engineering advancements can help us gain deeper theoretical insights and many efforts have been made [85; 1; 70; 2; 94] towards this goal.
To model the patterns of the signal spaces.
A central challenge of modeling the signals is the formalization of a small set of ideas for constructing the representations of the patterns[43; 79].
Wavelet transform[72; 24] was formulated to handle the sparsity in the natural signals.
Steerable and shiftable filters[38; 101; 99] were introduced to handle the discontinuity of wavelet filters with respect to the natural transformations.
Basis pursuit[17; 16] finds decomposition of a signal by a predefined dictionary, sparse coding[81; 82], and ICA[7; 56] were proposed to learn the filters or dictionary from data.
Before the sparse manifold transform[21], many important works have paved the way and introduced topology and transformations between the dictionary elements[57; 58; 11].
There are also several important works on studying the natural image patch space with non-parametric methods and topological tools [67; 26; 12; 48].
Manifold learning[89; 104; 44] and spectral clustering [33; 37; 95; 75; 80; 103; 93] was proposed to model the geometric structure in the signal space.
The geometric signal process is also an increasingly important topic in signal processing [96].
Modeling of biological visual system.
Exploring the geometric or statistical structure of natural signals are also strongly motivated by computational modeling of biological visual system.
Many properties of the visual system reflect the structure of natural images [100; 59].
An early success of this quest was that the dictionary elements learned in an unsupervised fashion closely resemble the receptive fields of simple cells in the V1 cortex[81; 82].
In order to further understand the visual cortex, we need to develop theories that can help to elucidate how the cortex performs the computationally challenging problems of vision [83].
One important proposal is that the visual cortex performs hierarchical manifold disentanglement [31; 32].
The sparse manifold transform was formulated to solve this problem, which is supported by a recent perceptual and neuroscience evidence [49; 50].
There are also several related seminal works on video linearization [111; 41; 40].
To build representation models from the first principles.
The now classical models before deep learning use to be relatively simpler [64; 107; 87; 60; 110; 114; 61], where sparsity was a widely adopted principle.
While the gap between the SOTA deep learning model and the simple models is large, it might be smaller than many expect [23; 88; 10; 105].
The quest to build “white-box” models from the first principles [14; 13; 71] for unsupervised representation remains an attractive open problem.
D ISCUSSION
In this paper, we utilize two classical unsupervised learning principles, sparsity and spectral embedding, to build a minimalistic unsupervised representation with the sparse manifold transform [21].
The constructed “white-box” model achieves non-trivial performance on several benchmarks.
Along the way, we address several fundamental questions and provide two complementary points of view, manifold disentanglement and co-occurrence modeling, of the proposed model.
Further, the proposed method has close connections to VICReg [5] and several other SOTA latent-embedding
SSL models [18; 116; 39].
There are several limits of this work, which point out the future directions: 1) We need to scale the method to larger datasets, e.g. ImageNet.
While a two-layer network achieves non-trivial performance, we need to further construct “white-box” hierarchical models to understand how a hierarchical representation is formed [51; 53].
While spectral embedding is context-independent [86; 117], there is clear benefit in building contextualized embedding [29; 91; 52; 36; 115; 73].
It is an attractive topic to build minimalistic unsupervised representation based on contextualized embedding.
The evidence shows that it is a promising direction to build minimalistic “white-box” unsupervised learning models.
Further reducing the gap and understanding why can lead us towards a theory.
Minimalistic engineering models are easier to analyze and help us build a theory to identify optimality and efficiency.
On the other hand, a testable theory can also facilitate the modeling of biological learning systems.
This work makes a step towards this goal.
Appendix A P ROBABILISTIC POINT OF VIEW EXAMPLE : SMT WORD EMBEDDING
We use SMT to build word embedding for the vocabulary in WikiText-103 to illustrate the probabilistic view of SMT.
Unlike in Figure 2, where the neighbor of each data point x can be interpreted as its Euclidean neighborhood, there is no such distance function in the natural language.
Nevertheless, we could still define neighborhood for each word.
We define the neighbor of each word as other words that co-occur with it in a given context.
Like the manifold learning view, where the global geometry emerges despite only local neighborhoods being used during training, such a phenomenon is also observed from the probabilistic point of view.
To visualize this phenomenon from a probabilistic point of view, we measure the cooccurrence of each pair of words and the cosine similarity between them in their SMT embedding space.
In figure 5 (a), we show the top 10 words that are closest to “raccoon” and the top 10 words that co-occur the most with “raccoon”.
We can see words like “American,” and “population” cooccur with “raccoon” a lot, but neither “American” nor “population” are close to “raccoon” in the embedding space.
Instead, words that are semantically related to “raccoon” are close to “raccoon” in the embedding space, like “squirrels”, “rabbits” and “wolves”, even though they rarely co-occur with “raccoon.”
The underlying reason is that these words share the same context patterns as “raccoon”.
Although words like “American” and “population” can co-occur with “raccoon” in phrases like ”American raccoon” and ”raccoon’s population,” these words also co-occur with other words in very different contexts.
Thus, they could not be embedded to be close “raccoon.”
We also illustrate three more examples in figure 5.
In (b), we see words that occur with “feburary” are other months like ”july,” ”june,” and ”september.”
In (c), words co-occur with “seattle” the most are other large cities like “dallas”, “denver” and “pittsburgh”, even though they rarely occur with “seattle.”
Finally, in (d), the words co-occur with “aspirin” the most are names of other medicines.
B D ETAILS FOR TRAINING WORD EMBEDDING USING SMT
Dataset.
We use WikiText-103 corpus [76] to train SMT word embedding, which contains around 103 million tokens.
For the tokenizer, we use “Basic English Tokenizer” from torchtext, which uses minimal text normalization.
We tokenize and lowercase the WikiText-103 corpus, building a vocabulary using the top 33,474 words with the highest frequency.
Notice that this dataset is significantly smaller than the Wikipedia dump used to train GLoVe in the original paper [86].
Wikipedia is also highly biased toward technical subjects, which is reflected in our word embedding.
SMT training.
First, we choose the sparse extraction function f to be the one-hot encoding.
Each token in the text corpus is considered a data point x i ; it is first transformed into a one-hot vector by f , then encoded to be close to its neighbor via spectral embedding.
In order to perform spectral embedding, we need to define the neighbor(context) for each word in Optimization 1.
In this case, each word’s neighbor(context) is its previous eight words and the next eight words.
We can then calculate ADD T A T , V , and solve Optimization 1 for P .
The representation of each word in the vocabulary is simply each column of P. This is because f is one-hot encoding, for each word ⃗x i , we have ⃗β
i =
Pf ( ⃗x i ) =
P e i
= ⃗P
i .
C W HITENING OF THE IMAGE PATCHES
We consider all the square image patches with size T × T × c , where T is patch size, and c is channel size, as flattened vectors of dimension T 2 c .
We first remove the mean from each patch.
Instead of calculating the global mean like in the normal whitening procedure, we calculate the contextual mean.
Let x i denote a image patch and let N i = { x j : j ∈ n ( i ) } denote the neighboring patch for x i .
We can calculate the contextual mean of x i as µ i = 1 ||
N i || (cid:80)
j ∈ N i x j .
Let ¯ x i denote the mean removed image patch, then ¯ x i
= x i − µ i .
We could also simply calculate the global mean µ and use it to center all the image patches.
However, we found using contextual mean improve performance.
We apply the whitening operator to the mean-removed image patches.
Let { ⃗x i } denote the preprocessed image patches.
Then we have
[Equation]
To “unwhiten” the patch for visualization, we multiply the whitened patch with the inverse of whiten operator: ( λI + Σ) 12 .
D 1-S PARSE D ICTIONARY L EARNING
Given ⃗x i ∈ R
d to be a data vector, and dictionary (codebook) Φ ∈ R d × K , where d is the dimension of the data and K is the number of dictionary elements, 1-Sparse Dictionary Learning, also known as Vector Quantization (VQ problem [42]), can be formulated as:
[Equation]
This is an NP-hard problem, but there are efficient heuristic algorithms to approximate the solution.
The most well-known one is K-mean clustering, which can be interpreted as an expectationmaximization (EM) algorithm for mixtures of Gaussian distributions.
This is the algorithm we used to solve 1-sparse dictionary learning when we use the 1-sparse feature f vq in SMT.
The exact algorithm we used to solve the 1-sparse dictionary learning problem is in Algorithm 1.
Notice that when data is uniformly distributed on the domain, even a dictionary consisting of uniformly sampled data point serve as a good enough approximation.
This is the case for the toy example in Figure 2.
In this case, the dictionary Φ can be predefined instead of learned.
For MNIST and CIFAR-10 data, they are both assumed to be low dimensional data manifolds in high dimensional space.
Thus, we cannot apply k-mean directly to the raw signal due to the high dimensionality and sparsity of the data.
One way to alleviate this problem is to map the data to a unit sphere before applying k-mean.
This method is called spherical k-mean.
More detailed motivations and advantages of this approach are explained in the original paper [30].
The exact algorithm is also described in Algorithm 1.
Algorithm
Data: learning
Result: Dictionary (centroid) Φ =
[ ⃗ϕ 1 , · · · , ⃗ϕ K ] , where each column is a dictionary element
Sparse code (label)
A =
[ ⃗α 1 , · · · , ⃗α N ] for each data point.
1 initialization: Φ ← 0 , A ← 0 ; 2 if spherical then for i ∈ 1 , · · · , n do ⃗x i = ⃗x i || ⃗x
i || end 6 end 7 while not ϕ has not converge do
Find code for every data point ; for i
, · · · , n do
[Equation]
∗
[Equation]
end
Dictionary update ;
[Equation]
; if spherical then do
end end 19 end
E A BLATION STUDY : FEATURE DIMENSION AND EMBEDDING DIMENSION
Since SMT is a white box model, we can analyze how each component of SMT contributes to unsupervised learning.
Specifically, we break down SMT into two components: dimensionality expansion using sparse features and dimensionality reduction through spectral embedding.
We first show that a sparse feature by itself is important in building a good representation.
As shown in Figure 6, we’re able to achieve 71 % evaluation accuracy using only general sparse features f gq on the CIFAR-10 dataset. Please refer to Section 3 for the definition of general sparse features f gq .
Combining sparse features with whitening will further boost the performance to 75 %.
However, the performance does not increase as the number of features increases if we’re using only sparse features and whitening.
In contrast, if we combine sparse features and spectral embedding, we do see the performance increase as the number of features increases.
This result implies that spectral embedding is also a crucial element in obtaining a good representation.
To further show the necessity of spectral embedding, we did an experiment to replace spectral embedding with PCA.
PCA is a dimension reduction algorithm
but does not enforce locality.
In Figure 6(b), we see that if we’re using PCA as dimension reduction, the performance decreases monotonically as we reduce the sparse feature to a lower dimension, which implies it is throwing away useful information.
On the other hand, for SMT, the performance first increases and then decreases, as we increase the reduction dimension.
The performance of SMT peaks when we use 500 as the reduction dimension, which is much higher than PCA (79.4% vs 59.9%) with 500 components and even higher than PCA with all principle components (79.4% vs 75.2%). F H YPERPARAMETER FOR PRETRAINING DEEP SSL MODEL For all the benchmark results on the deep SSL model, we follow the pretraining procedure used in solo-learn. Each method is pretrained for 1000 epochs.
We list the hyperparameter we used to pretrain each model.
For both methods, we use resnet-18 as the backbone, and a batch size of 256,
LARS optimizer, with 10 epochs of warmup follows by a cosine decay.
For the SimCLR model, we set the learning rate to be 0.4 and a weight decay of 1e-5.
We also set the temperature to 0.2, the dimension for the projection output layer to 256, and the dimension of the projection hidden layer to 2048.
For VICReg, we set the learning rate to 0.3 and a weight decay of 1e-4.
We also set the temperature to 0.2, and set the dimension for the projection output and hidden layer to 2048.
We set the weight for similarity loss and variance loss to 25, and the weight for covariance loss to 1.
We use the transform module in pytorchvision to implement data augmentations.
By default, we use “RandomResizedCrop” with min-scale equal to 0.08 and max-scale equal to 1.0.
We also use “RandomHorizontalFlip” and use 0.5 as the flip probability.
For “Grayscale Image Only”, we apply the “RandomGrayscale” transform with the grayscale probability to 1.
For “Original + Grayscale,” we set the grayscale probability to 0.5.
For “Full” color data augmentation, we add “ColorJitter.”
For SimCLR, we use brightness = 0.8, contrast= 0.8, saturation=0.8, and hue=0.2 as its “ColorJitter” parameter.
For VICReg, we use brightness = 0.4, contrast= 0.4, saturation=0.2, hue=0.1 as its “ColorJitter” parameter.
We also add additional “Solarization” transform with probability 0.1 for training VICReg.
G A BLATION STUDY : CONTEXT RANGE
Given an image patch { ⃗x ij } , we define its neighbors to be other patches within a specific pixel range from it on the same image, i.e., n ( ij ) = { lm ; | i − l | ≤ d and | j − m | ≤ d } .
We use d to denote the “context range”.
We showed the impact of context range on MNIST in table 1.
The performance of
SMT decreases as we increase the context length.
We also studied the effect of context length on the
CIFAR-10 dataset and showed the result in table 5.
Unlike MNIST, the performance on CIFARdataset increases as the context window increases.
This once again confirms that both the manifold view and the probabilistic co-occurrence view are necessary to understand SMT.
For CIFAR-10, the images have lots of high-level structures.
The key to building a good representation is the ability to model the co-occurrence between different image patches.
For example, in an image of an animal,
[Table]
[Table]
the animal’s “leg” and “head” might lie in two image patches far apart.
But it is very important to capture the fact that “leg” and “head” co-occur, because this is a unique trait for animals, which requires the context range to be large.
This also requires the probabilistic co-occurrence view.
Many image patches can be neighbors (co-occur)
[Table]
but they cannot all be encoded to the same point.
Instead, if we record the co-occurrence, patches that co-occur more often should be encoded closer than to each other.
On the other hand, MNIST is closer to a manifold, so co-occurrence is less informative in this case.
Modeling co-occurrence between image patches might result in overfitting.
In order to form a good representation, we need to use a small context range to capture slow features.
This is a lot like the formulation in the original sparse manifold paper, where they use consecutive video frames to define context, which is essentially a context range of only 2 pixels.
H A BLATION S TUDY : NUMBER OF TRAINING EXAMPLES
We experiment with the impact of a number of training examples on performance.
The experiment is done on the CIFAR10 dataset.
The result is shown in table 6.
The result shows SimCLR experienced a much more significant performance drop compared to SMT when training with a limited amount of data.
The significant performance drop of the deep SSL method is likely due to the well-known overfitting problem of complicated models like neural networks.
This shows the advantages of SMT as a simple white-box model.
I A BLATION STUDY : LINEAR PROBING VS KNN FOR EVALUATION
We experiment with using linear probing for evaluation instead of the KNN classifier.
We also experiment with the comparison between linear probing and KNN classifiers with a different number of embedding dimensions.
As shown in the table 7, with optimal embedding dimension, linear probing accuracy is higher than KNN accuracy (81.0% vs 79.4% ).
This implies that the linear layer learned in SMT is not optimal for image classification, but it is still a pretty good representation for image classification purposes.
Most useful information for image classification is captured during this unsupervised dimension reduction process.
This explains why the optimal embedding dimension is much higher (3200) when using a linear classifier compared to when using a KNN classifier (500).
J S OFT KNN E VALUATION
Classic K-nearest-neighbor classifier (hard KNN): for each data point x , which is labeled with k class, let y, y j denote the label of x, x j .
Then let ˆ z ( x ) = 1 | n ( x ) | (cid:80) x j ∈ n ( x ) OneHot ( y j ) , which counts the average number of time x ’s neighbors belong to each class.
And q ( x ) = softmax ( z ( x )) , i.e. q ( x ) k
= exp ( z ( x ) k T ) (cid:80) l exp ( z ( x ) l T ) .
This quantity is the estimated conditional probability that the data point
[Table]
x belong to k th class, given temperature T , i.e. P ( y = k | x ) .
Finally, we simply assign the class with largest conditional probability as the predicted class label i.e. ˆ y ( x ) =
arg max k q ( x ) k .
Soft K-nearest-neighbor classifier (soft KNN): for a given data point x , instead of using only x ’s neighbors’ class label to estimate the conditional probability, we also we take into account of the cosine similarity between x to its neighbors.
Instead of having ˆ z ( x ) =
[Table]
n
[Table]
x ) |
[Table]
OneHot (
[Table]
[Table]
we have ˆ z ( x ) = 1 | n ( x ) | (cid:80) x j ∈ n ( x ) OneHot ( y j ) cos ( x j , x ) .
The rest of the algorithm is exactly the same as hard KNN.
In all the experiment, we set K = 30 and T = 0 .
03 for evaluation.
K C ONNECTION TO DEEP LATENT - EMBEDDING SSL METHOD
The general formulation of SMT, as stated in Section 3, is:
[Equation]
,where A denotes the sparse representation of the data.
Feature extraction: As discussed in Section 2, sparse features are used to capture locality and compositionality.
There’s also other feature that might be used to serve the same purpose.
We can abstract such feature extractor as a function f : X → A .
Projection: As discussed in Section 2, we assume the sparse feature of the data is a low-dimensional manifold that lies in a high-dimensional space.
Thus, we seek an optimal map that projects the sparse representation to the low dimensional space, such that points which are close in feature sparse should be close after the projection as well.
In our setup, we use a linear function to approximate this optimal projection function, but the projection function doesn’t have to be linear in general.
We can abstract this projection function as a function g : A → Z .
Let Z =
g
( f ( X )) , then SMT objective can be generalized to the following form:
[Equation]
Consider the objective for VICReg:
[Equation]
where Z = g ( f ( X )) , and both feature extractor f and projector g are parametrized by neural networks.
Like discussed in [4], we use a friendlier variance loss: replacing the hinge loss at 1 as square loss.
We also assume α = β = γ = 1 for simplicity.
Then we have:
[Equation]
Thus
[Equation]
This is the same objective as Eq 1.
We can construct D as the same first order derative operator as the one described in Section 2, then we have:
[Equation]
With generalized features and projects, we’re able to formulate VICReg as a sparse manifold transform problem.
In VICReg, since both parameters f and g are parametrized by neural networks, there is no close-form solution to VICReg.
They are optimized with a stochastic gradient descent-based algorithm.
As discussed in [4], most SSL models can be reduced into the same form, which means they can all be formulated into SMT objectives.
L I NTERPRETING SIMILARITY LOSS AS FIRST - ORDER DIRECTIONAL DERIVATIVE
Similarity can be considered as a first-order directional derivative of the representation transform w.r.t.
the similarity neighborhoods.
A representation transform g maps a raw signal vector ⃗x into a representation embedding ⃗β , i.e., g : R m → R d .
In SMT, g = Pf ( · ) , where f is a sparse feature function.
The definition of directional derivative is: D ˆ u g ( α ) =
lim h → 0 g ( α + h ˆ u ) − g
( α ) h ,where ˆ u is the direction we want to take the derivative and h is the infinitesimal change.
In a discrete space, we represent the dataset as a graph.
Each data point is a node that connects to its neighbors.
As we discussed at the beginning of the paper, “similarity” neighborhoods can be defined by 1) temporal co-occurrence, 2) spatial co-occurrence, 3) local metric neighborhoods in the raw signal space.
The direction between two neighboring raw signal points x i and x j is defined as their direction in the sparse feature space, i.e., f ( x j ) − f ( x i ) .
Unlike in the real coordinate space, where we can take infinite direct directions at each location, with the discrete setting, we can only take a finite number of directions at each data point (the same number as the number of its neighbors).
Moreover, since everything is discrete, the infinitesimal change h = 1 .
For example, we want to take the directional derivative at point x i and in the direction towards its neighbor x j .
Then we have ˆ u = f ( x j ) − f ( x i ) , and
[Equation]
M V ISUALIZATION FOR MISCLASSIFIED EXAMPLE FOR CIFAR-
We provide the visualization for examples misclassified by our SMT + KNN model.
In Figure E, the figure is divided into two panels, with 25 images on the left and 24 images on the right.
In the left panel, we show a image being classified on the top left corner and its 24 nearest neighbors.
In
Figure E, the image being classified is a image of a “dog”.
Almost all of its 24 nearest neighbors are “horse.”
As a result, the “dog” image is misclassified as a “horse”.
The right panels show the patches with highest cosine similarity to their corresponding patch in each of the 24 nearest neighbors.
One insight we can learn from these visualizations is that these misclassifications are largely due to local-part matching.
For example, a “deer” may look like a “horse” except for the “deer horn” parts.
However, “deer horn” parts may only contribute a small fraction in the similarity measurement.
As a result, the nearest neighbors of a “deer” may be almost all “horses”.
To solve this issue, again, we need to have a better understanding of hierarchical representation learning and high-level concept formation.