U NSUPERVISED C LASSIFICATION INTO U NKNOWN NUMBER OF CLASSES
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel spectral decomposition framework for the unsupervised classiﬁcation task.
Unlike the widely used classiﬁcation method, this architecture does not require the labels of data and the number of classes.
Our key idea is to introduce a approximate linear map and a spectral decomposition method on the dimension reduced space into generative adversarial networks.
Inspired by the human visual recognition system, the proposed framework can classify and also generate images as the human brains do.
We build a approximate linear connection analogous to the cerebral cortex, between the discriminator D and the generator
G .
This connection allows us to estimate the number of classes k and extract the vectors that represent each class.
We show that our framework has the reasonable performance in the experiment.
I NTRODUCTION
Image classiﬁcation has been one of the most important research in machine learning and artiﬁcial intelligence.
For the predeﬁned number k of classes, the classiﬁcation models based on deep learning have performed well that the test accuracy is better than humans (Geirhos et al., 2017;
He et al., 2016; Russakovsky et al., 2015; Szegedy et al., 2015).
This result is, however, true only when the number of classes, k is predeﬁned and we use the supervised learning with entire ground truth labels.
Humans learn visual information by memorizing the compressed information (features) in the brain, reconstruct the object, and recognize the difference between objects.
This capability allows humans to classify data where the number of classes is unknown into the reasonable numbers of classes.
In this paper, we propose a method to mimic such human capability.
The main contribution of the proposed framework consists of two parts.
We ﬁrst show that there exists a approximate linear map between two separated neural networks of Generative Adversarial
Networks (GANs).
Secondly, we prove that the number of classes in a latent space is the same as the number of classes in the original space in terms of the multiplicity (corresponding to eigenvalue 0) of a Laplacian matrix in spectral clustering theory.
By combining the results, We derive a novel unsupervised classiﬁcation framework, which estimates the number of classes k and then extract k vectors representing each class.
With the framework, the approximate linear map between two networks can induce the information of the spectral decomposed components in a data graph.
The latent variable model is commonly used in the generative models.
Variational auto-encoders (VAEs) (Kingma & Welling, 2013) and GANs (Goodfellow et al., 2014) are a few of the most famous cases that learn the data distribution successfully.
GANs randomly samples the latent variables that are used to train the data distribution so we cannot ﬁgure out which latent variable generates a result that we are interested in.
To avoid the drawback by random sampling, auxiliary conditions on latent variables were introduced (Chen et al., 2016; Mirza & Osindero, 2014).
Introducing a label information into the condition of the estimated data probability is highly effective in classiﬁcation where a data point x came from, however this method is not a unsupervised learning.
In addition, there are some GANs frameworks that analyze the property of latent spaces (Kumar et al., 2017;
Radford et al., 2015).
Spectral clustering is one of the most popular clustering algorithm that outperforms other wellknown clustering algorithms (Ng et al., 2002; Shi & Malik, 2000).
Despite high performance, spectral clustering is difﬁcult to use because it requires the full dataset that have very large N data points.
In this case, the dimension of the Laplacian matrix is N × N and it takes at least O ( N 3 ) .
Therefore, we need a novel dimension reduction technique to use spectral clustering in practical applications (Elhamifar & Vidal, 2009; Ji et al., 2017; Law et al., 2017; Shaham et al., 2018).
Especially, we apply the self-expressiveness property (Elhamifar & Vidal, 2009) to the proposed method.
The main concepts of spectral clustering, GANs and the self-expressiveness property will be introduced Section 2.
We prove the preliminary propositions that allow us to classify data with unsupervised learning methods in Section 3.
Finally, we propose a unsupervised classiﬁcation algorithm and test our algorithm in Section 4.
Recent image classiﬁcation methods are based on supervised learning, in which a learning model processes a dataset consisting of pairs of an image and its ground-truth label.
This training dataset usually consists of more than 10 K data point, some part of which need to be used for crossvalidation.
It also takes a lot of costs to produce a reliable training dataset.
P RELIMINARIES
In this section, we review cycle-consistent adversarial networks and sparse subspace clustering that we will use when extracting indicator vectors.
We also overview the basic probabilistic property between a classiﬁcation model and a generative model, which can be used as tools to generate indicator vectors.
2. CYCLE - CONSISTENT ADVERSARIAL NETWORKS
Generative Adversarial Networks (GANs) is a generative model that a generator network learns a data distribution implicitly (Goodfellow et al., 2014).
GANs is a two player minimax game with two neural networks: a discriminator D and a generator G .
The discriminator D ( x ) is a probability measure that an input x follows the data distribution p data ( x ) rather than the generator’s distribution p g .
The generator G is a mapping from the latent space z ∼ p z to the data space.
This minimax game can be represented as a stochastic optimization problem with value function given by
[Equation] [Equation]
[Equation]
The cycle-consistent adversarial networks (cycleGANs) is a variant of GANs, which learns how to translate an representation from a source domain S to a target domain T without a paired dataset (Zhu et al., 2017).
The cycle-consistency states that two mappings G : S → T and F : T → S encode the right permutation of images in S and T , so that, for an arbitrary data point s ∈ S , the mapping from S to T and back again, F ( G ( s )) , should be the same as s .
This property encourages adversarial networks G and D to have a connection between generated images and input data.
We adopt two cycle consistency loss for the forward map and the backward map to learn a connection between two high-level representation domain in G and D .
2. S PARSE S UBSPACE CLUSTERING
Sparse subspace clustering is a clustering algorithm that is based on spectral clustering.
Spectral clustering has many variants depending on the simiiliarity measure, Gaussian kernel similarity function (Ng et al., 2002), k -nearest neighbor graphs, (cid:15) -neighborhood graph.
A similarity measure deﬁned on the similarity graph G =
( V, E ) is a pairwise metric between two data points (nodes) in a dataset and the similarity w ij of two data points x i , x j is equal to or greater than 0 .
The similarity w ij is also same as w ji because the above similarity graph is deﬁned as an undirected graph.
The similarity matrix W is composed of the elements which correspond to the similarity measure w ij respectively and then W is symmetric.
We can deﬁne the graph Laplacian matrix of the similarity graph G as L = D − W , where D is the degree matrix and deﬁned as a diagonal matrix that the diagonal elements is given by d i = (cid:80) j w ij .
The multiplicity k of the eigenvalue 0 of L is the number of connected components
A 1 , · · · , A k in the similarity graph.
Each connected component is a partition of the similarity graph G and considered as a subgraph, i.e., a cluster.
See more details in (Von Luxburg, 2007).
The proper design of weight measure w : X × X → R + is a key to clustering data very well because the graph Laplacian matrix L is based on a similarity matrix W .
Sparse subspace clustering (Elhamifar & Vidal, 2009) uses a novel similarity matrix that is based on the self-expressiveness
[Figure]
property.
When we assume that the data space is a union of linear subspaces S = ∪ kl =1 S l , we can express an arbitrary data point in the dataset as a linear combination of other data points in the dataset, where the coefﬁcients of the data points in the same subspace are non-zero and others should be 0 (Elhamifar & Vidal, 2009; Ji et al., 2017).
This property is called self-expressiveness property.
Formally, each data point y i ∈ S can be reconstructed as y i
=
(cid:80) Nj =1 c ij y j
= Yc i , where c ii = 0 , N is the size of the dataset, c i :=
[ c i 1 c i 2 · · · c iN ] T is the coefﬁcient vector of y i and
Y =
[ y 1 y 2 · · · y N ] is the dictionary matrix.
The sparse solution of c i indicates that the node i is in the same subspace as the node j if the element c ij is non-zero.
In this paper, we expand the notion of the self-expressiveness property to reduce the dimensionality of L in Section 3.3.
G RAPH L APLACIAN OF THE CONNECTOR NETWORK
We ﬁrst introduce our proposed architecture of adversarial networks.
As point of view as autoencoder, we can consider two adversarial networks as autoencoder.
We build a connector network between two networks to improve the cycle consistency and extract the information of connected components.
(As shown in Figure 1-b).
By training with the cycle-consistency loss, we can make the tuples ( Z, X, H ) for each data point X in a given dataset.
Then, we show the relation between the spectral clustering and unsupervised classiﬁcation.
In order to show the relation, we prove the dimension reduction of the mutual expressiveness property.
3. D ISENTANGLING UNFOLDS AND E XPANDING THE VOLUME OF MANIFOLD Deep
neural networks learn the high-level abstract representation of the data manifold and the well trained abstraction can disentangle the underlying factors of variation (Bengio et al., 2007).
These properties allow interpolating between the data samples more naturally and smoothly.
(Bengio et al., 2013) studied the disentangling effect and proposed three hypotheses about the shape of manifolds in the different representation domains.
These hypotheses are veriﬁed by several empirical implementations (Goodfellow et al., 2009; Glorot et al., 2011).
We can summarize three hypotheses as follows.
• The deeper representations can better disentangle the underlying factors of variation.
[Figure]
• Disentangled representations unfold the manifolds near which raw data concentrates irregularly and expand the relative volume occupied by high-probability points near these manifolds.
Then these representations are with greater convexity.
• The underlying class factors of variations will be better disentangled than other factors, so the deeper layer has the better discriminant ability of classes.
In our proposed architecture, there are two high-level (deeper) representation spaces HandZ which is called as feature spaces.
These feature spaces ﬁll more uniformly the space than the pixel space
X .
H and Z also have the smoother density.
Therefore, H and Z have greater than convexity than
X
and it allows the linear combination on these two manifolds in the feature spaces H and Z are and be more convex than the manifold in the pixel space X .
By manifold hypothesis, the different class manifolds are well-separated by regions of very low density (Cayton, 2005).
The feature spaces are the high-level representation spaces so that the density are unfolded and the linear combinations of features generates more natural results in the pixel space Bengio (2014).
Now, we can deﬁne the effective basis of the subspace for each class.
In the feature spaces, each class manifold has the enough convexity.
The class manifolds are well-separated and show good discriminant ability.
Using both properties, we can build the effective basis for each class subspaces and generate the original space by the direct sum of these subspaces, where the subspaces are orthogonal.
We empirically tested the effective basis as shown in Figure 3.
Randomly sampled features in a subspace of the data space can be an effective basis and be possible to generate all data point in the same subspace.
The features in the different subspaces cannot generate the given data samples.
We can estimate the effective dimension using the number of the effective basis.
Then, we can generate orthogonal subspaces of the entire classes if the effective dimension is small enough than the dimension of feature spaces.
3. C ONNECTOR NETWORK AND CYCLE CONSISTENCY
We will study the relation between 3 different spaces, Z , X and H in the proposed architecture.
As shown in the ﬁgure 1-(b), we can consider two adversarial networks as a decoder and an encoder of an auto-encoder respectively
(Berthelot et al., 2017; Zhao et al., 2016).
The discriminator network
D encodes the pixel space X to the feature space H and generator network G decodes the feature space Z to X .
We deﬁne the connector network C which maps from H to X .
The map C between two feature spaces will be trained by the cycle consistency loss to obtain the tuples ( Z, X, H ) with the correct permutation, where all of elements is in the same-class manifold and shares same learned features.
Cycle consistency loss tends to minimize the difference between an arbitrary point and the returned point after traveling other representation domains.
This cycle-consistency objective can allow the inverse maps between all representation spaces.
When we apply the forward cycle-consistency loss (cid:107) z − C
( D ( G ( z )))
(cid:107) 1 and backward cycle consistency loss (cid:107) x − G ( C ( D ( x )))
(cid:107) 1 , the maps between all two spaces become unique.
Therefore, we will induce the feature-invariant one-to-one maps
[Figure]
G, C, D .
All elements of an arbitrary tuple ( Z, X, H ) share same class features as shown In ﬁgure 1-(a).
Therefore we now naturally build the following assumption.
Assumption 3.1.
All elements of the cycle consistent tuple ( Z, X, H ) shares the identical factors, such as class factor and other factors (e.g. rotation and brightness).
As shown in ﬁg 3-(c, d), we can conjecture that each class subset is closed and the linear combinations in the certain class subset are also in the class because each class manifold is unfolded and expanded in Z and H .
In addition, we showed there exists the effective basis for each class.
By the hypotheses (Bengio et al., 2013), we know that each class manifold has the compactness and the convexity.
Now we can assume the following assumption.
Assumption 3.2.
Each class subspace H i , where H i ⊂ H are orthogonal, so as Z i where Z i ⊂ Z .
We deﬁne Sets for each class S H i and S Z i in H i , Z i respectively to be the the class manifolds which lies on the corresponding class subspaces.
The sets S H i and S Z i are compact and convex.
This assumption is more tight than the union of linear subspaces in Elhamifar & Vidal (2009).
We can also conjecture that the feature space H and Z are descried as the direct sum of H i and Z i respectively.
Now, we can show the existence of an approximate linear map C from H to Z .
Lemma 3.1.
There exists a compact and convex linear map C i from S H i to S Z i , If (cid:107) C i (cid:107) is bounded.
Proof.
By the asumption 3.2, the class subset
S H i and S
Z i are convex.
Let h 1 , h 2 ∈ S H
i and z 1 = C i h 1 , z
2 = C i h 2 .
We know that C i is linear, then
[Equation]
th 1 + (1 − t ) h 2 is also in S H
i for t ∈ [0 , 1] .
Also, tz 1 + (1 − t )
z 2 is in S Z i .
Therefore C i is convex.
If S H i is compact, then for h ∈ S H
i , (cid:107) h (cid:107) is closed.
Then z ∈ S Z i is closed, because
[Equation]
We know that C i and h have ﬁnite values.
Theorem 3.1.
By the assumption that the data space are the direct sum of orthogonal class subspaces, the map from H to Z can be written as a block matrix, which block component is C i .
C can maps between all class subset approximately.
Proof.
C i is a linear map from H i to Z i .
H i
and Z i are orthogonal.
It is trivial.
3. M UTUAL EXPRESSIVENESS PROPERTY AND DIMENSION REDUCTION
As we discussed in Section 2, the self-expressiveness property shows that we can express an arbitrary data point as a linear combination of other N data points.
Modifying this property slightly, we can write each data point x as a linear combination of N sampled data points α = { x 1 , x 2 , · · · , x N } from the data space X .
This fact allows us to generate a set of feature variables, which satisﬁes the self-expressiveness property.
The self-expressiveness property implies that the generated data points are in the class manifolds.
Now, we weaken the self-expressiveness property to generalize the situation of the problem.
We divide the latent space H into H 1 , · · · , H l where the self-expressiveness only satisﬁes in each H i not in H − H i .
We also know that it is possible to discard redundant coefﬁcients in the self-expressiveness expression by optimizing the matrix of coefﬁcients (Elhamifar & Vidal, 2009), so there exists at least one sparse solution of x where most coefﬁcients are zero.
By observing this property, we can expect that the number of non zero coefﬁcients goes to the dimension of latent space, m .
That is because
N is much higher than dim X and X is a m -dimensional manifold.
Image clustering methods, for instance, treat the dataset in which the number of data points is much larger than the dimension of data points so we can optimize a solution to have sufﬁciently enough zero coefﬁcients in the self-expressive linear combination.
Theorem 3.2.
From now on, Z (cid:48) i = C (cid:48)
i H (cid:48)
i where Z (cid:48)
i =
( z 1 , · · · , z m i ) T and H (cid:48)
i =
( h 1 , · · · , h m i ) T .
Proof.
By the observation on GANs, we have N i feature variables β
i = { h ij } 1 ≤ j ≤ N i for each 1 ≤ i ≤ l where D ( x i ) =
h i and N = N 1 + · · · + N l .
By the self-expressiveness property in H i , we can express each h
i ∈ H
i by the linear combination of β (cid:48)
i = { h 1 , h 2 · · · , h m i , h
m i +1 , · · · h N i } where β (cid:48)
i is a reordered set of β
i and the coefﬁcents of the linear combination { c m i +1 , c
m i +2 , · · · , c Ni } are zero.
In other words, the ordered set γ i = { h 1 , · · · , h m i } is enough to represent all points in the class manifold X i and Z i , which correspond to ( Z i , X i , H i ) , i.e. span ( γ i )
=
H i .
Hence, we can choose C (cid:48)
i by an m i × m i real matrix.
This dimension reduction has a crucial role in reducing the dimension of Lapalacian matrix of which similarity matrix is | C | + | C T |
Theorem 3.3.
Suppose that G is an undirected graph with non-negative weights.
Then, the number of connected components A 1 , · · · , A k in G is same as the multiplicity k of the eigenvalue 0 of L .
The eigenspace corresponding to the eigenvalue 0 is spanned by the indicator vectors 1 A 1 , · · · , 1 A k of those components.
Proof.
the proof in (Von Luxburg, 2007).
3. U NSUPERVISED CLASSIFICATION
Now, we consider the unsupervised classiﬁcation problem that is not given even a single ground-truth label for the given dataset.
For example, when the cross-entropy loss is applied in the supervised classiﬁcation, we need pairs of input data and ground-truth probability mass p =
[0 , , 1 , · · · , 0] , which contains single 1 at the i -th element, i.e. the class index c
= i , to estimate the difference between the prediction and the ground-truth.
We can formally write the probabilistic interpretation as
[Equation]
, where x is a data point.
If we assume that the numbers of data points in the whole classes are same, then P ( c ) is uniform.
Then, for the given point x , P ( x ) is ﬁxed so the we can measure P ( c | x ) by the estimating the likelihood P ( x | c ) .
In Section 3.3, we showed that the indicator vectors of the connected components are the eigenvectors of the Laplacian matrix L , which are corresponding to the eigenvalue 0.
The components of the indicator vectors span the eigenspaces of the matched connected components respectively.
The orders of components for the Laplacian matrix L C and the connector network C are identical by the deﬁnition in Section 3.3.
Therefore, for all connected component
A i , the indicator vectors 1 A i = h ∈ H i spans the class subspace H i .
The indicator vector itself, however, might not be in the class manifold because it can exist outside the boundary of the manifold.
We ﬁnd out that the changes of norms between H and Z are different by the class.
The connector network can be considered as a singular value decomposition for each class.
This implies that each class data points move differently by the class.
We deﬁne the mean vector of a class in H as h (cid:48) and the Gaussian noise n .
[Equation]
The expectation of (cid:107) z (cid:107) 2 is,
[Equation]
, where n Z is the variation in Z .
We now assume that the distribution of (cid:107) z (cid:107) 2 is Gaussian.
Then, we can calculate the probability that (cid:107) z (cid:107) 2 is in the certain class.
We also know that B = C T C is a normal operator, then B satisﬁes the following theorem.
Theorem 3.4.
| x T Bx | ≤ max {| λ | : λ is an eigenvalue of B }(cid:107) x (cid:107)
Proof.
By ordering the eigenvalue of B , we can ﬁnd out the interval that (cid:107) z (cid:107) 2 lies in.
This eigenvalue works as the indicator of classifying class.
U NSUPERVISED CLASSIFICATION INTO UNKNOWN k CLASSES
In this section, we describe the loss function and the training algorithm of unsupervised classiﬁcation into unknown k classes.
Is is known that estimating the number of classes k is hard and heuristic (Von Luxburg, 2007).
We ﬁnd the empirical observation on the hint of k .
Using the observation, we suggest a conjecture that helps to estimate the number of classes k in the certain case.
1: for the number of iterations do 2:
Sample minibatch of m latent vectors z (0) , · · · , z ( m prior p g z
Sample minibatch of m data points x (0) , · · · , x ( m ) from dataset 4:
Update the discriminator by ascending : ∇ θ d 1 m m (cid:88) i = (cid:104)
[Equation]
Update the generator and connect by descending their stochastic gradients:
[Equation] [Equation]
Update the connector network C with two cycle-consistency loss
[Equation]
Compute the moving average of each i -th eigenvalue of normalized symmetric Laplacian matrix L sym = D − 12 LD − 12 and ﬁnd the estimated number ˆ k by ﬁnd the second deep local minimum 8: end for
4. ESTIMATING THE NUMBER OF CLASSES k
We found an interesting observation about the number of classes k .
The eigenvalue λ of the graph
Laplacian matrix equals to Ncut of the given graph (Shi & Malik, 2000).
In the ideal cases, we can assume that L is a block-diagonal matrix and
each of blocks L i is a proper sub-graph Laplacian.
If there is no connection between subgraphs, then L is a perfect block-digonal matrix and the multiplicity of the eigenvalue 0 is same as the number of blocks.
In the real world dataset, there exists quite large off-diagonal noise in the similarity matrix, as shown in the previous result.
These non-zero off-diagonal elements make the eigenvalue zero only at the ﬁrst eigenvalue and increase the Ncut of the graph gradually.
We can consider the i − th eigenvalue as the Ncut by the block diagonal matrix with i blocks, which is applied on the perturbed block diagonal matrix of which the underlying connected component is k .
Now, we conjecture the pattern of the eigenvalue.
We assume that the off-diagonal elements have the same value and the dimension of the all eigenspaces are same.
Now, we estimate the number of classes k by starting from m (cid:28) k .
We can consider the difference of Ncut between k -block diagnoal matrix and m -block diagnoal matrix as the the area of S m − S m ∩ S k , where S m = n 2 m and S k = n 2 k when the matrix size is n .
As m increases, the difference of Ncut increases, but at k = m the value have the very small local minimum becuase two matrices are perfectly overlapped.
Using this property we can derive the function of the difference of the Ncut over m, as follows.
[Equation]
where m ≤ k .
if m > k , swith m and k vice versa.
This overlapped area is n 2 k at m = k trivially.
The ﬁgure of the theoretical conjecture is in 4a.
This result can predict the emprical result of the pattern of the eigenvalue.
Therefore, we can conjecture that the underlying number of classes is at the second deep local minima.
E VALUATION
We introduce the setting of experiments and the architecture of our adversarial networks (Radford et al., 2015).
For the smaller size of dataset, such as MNIST and CIFAR10 we reduce the size of kernels properly.
We have the result the expectation of the estimated number of k is 10.2 for MNIST and 10.1 for CIFAR10.
Now we use the unsupervised clustering accuracy (ACC)
(Cai et al., 2011;
Shaham et al., 2018).
[Equation]
(13) where l i is true label,
c i is the ground-truth label, and π is the collection of permutations.
[Table]
[Figure]
Figure a shows theoretical result of the estimated eigengap using the overlapped area of blockdiagonal matrices and Figure (b) and (c) shows the eigengap | λ k − λ k − 1 | of the MNIST and CIFAR10.
C ONCLUSION
We have proposed an unsupervised classiﬁcation method that can automatically estimates the unknown number of classes k .
Our method provides the learned class-feature and this implies that the label of the given data point is automatically generated.