L EARNING L ATENT R EPRESENTATIONS IN N EURAL N ETWORKS FOR C LUSTERING THROUGH P SEUDO S UPERVISION AND G RAPH - BASED A CTIVITY R EGULARIZATION
A BSTRACT
In this paper, we propose a novel unsupervised clustering approach exploiting the hidden information that is indirectly introduced through a pseudo classiﬁcation objective.
Speciﬁcally, we randomly assign a pseudo parent-class label to each observation which is then modiﬁed by applying the domain speciﬁc transformation associated with the assigned label.
Generated pseudo observation-label pairs are subsequently used to train a neural network with Auto-clustering Output Layer (ACOL) that introduces multiple softmax nodes for each pseudo parent-class.
Due to the unsupervised objective based on Graph-based Activity Regularization (GAR) terms, softmax duplicates of each parent-class are specialized as the hidden information captured through the help of domain speciﬁc transformations is propagated during training.
Ultimately we obtain a k -means friendly latent representation.
Furthermore, we demonstrate how the chosen transformation type impacts performance and helps propagate the latent information that is useful in revealing unknown clusters.
Our results show state-of-the-art performance for unsupervised clustering tasks on MNIST, SVHN and USPS datasets, with the highest accuracies reported to date in the literature.
I NTRODUCTION
Clustering, the unsupervised process of grouping similar examples together, is one of the most fundamental challenges in machine learning research and has been studied extensively in different aspects such as feature selection, distance functions, grouping methods, etc.
(Aggarwal & Reddy, 2014).
k -means (MacQueen et al., 1967) and Gaussian Mixture Models (GMM) (Bishop, 2007) are two well-known conventional clustering algorithms that are applicable to a wide range of problems.
Traditionally, these methods are applied to low-level features such as raw data or gradient-orientation histograms (HOG) for images.
Therefore, their distance metrics are limited to local relations in the data space and inadequate to represent hidden dependencies in latent spaces.
On the other hand, spectral clustering (von Luxburg, 2007) is another conventional approach producing more ﬂexible distance metrics than k -means and GMM.
However, these types of solutions are not scalable to large datasets as they need to compute the full graph Laplacian matrix.
In recent years, researchers have focused on the unsupervised learning of high-level features on which to apply clustering and shown that learning good representations is important for the accuracy and robustness of the clustering task.
Deep Embedding Clustering (DEC) (Xie et al., 2016) was proposed to simultaneously learn feature representations and cluster assignments using deep neural networks (DNN).
In this approach, ﬁrst DNN parameters are initialized with a layer-wise trained deep autoencoder (Vincent et al., 2010) and then the initialized DNN is used to obtain the latent representation on which to perform k -means clustering for the initialization of cluster centers.
This complicated initialization is followed by a challenging optimization process that minimizes the Kullback–Leibler (KL) divergence between the centroid-based probability distribution and the auxiliary target distribution derived from the soft cluster assignments.
Similarly, Joint Unsupervised
Learning (JULE)
(Yang et al., 2016) combines agglomerative clustering with convolutional neural networks (CNN) and formulates them as a recurrent process.
Although JULE proposes an end-to-end learning framework, it suffers scalability issues due to its agglomerative clustering.
Novel deep generative models that can be trained via direct backpropagation have recently been proposed avoiding the difﬁculties in preexisting generative models such as Restricted Boltzmann
Machines (RBM), Deep Belief Networks (DBN) and Deep Boltzmann Machines (DBM) that are trained by MCMC-based algorithms (Hinton et al., 2006; Salakhutdinov & Hinton, 2009).
Among two canonical examples of these models, Variational Autoencoders (VAE) (Kingma & Welling, 2013; Rezende et al., 2014) integrate stochastic latent variables into the conventional autoencoder architecture while Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) propose an adversarial training procedure implementing a min-max adversarial game between two neural networks: the discriminator and the generator.
Following these advances, researchers have started to study new hybrid models with the goal of performing unsupervised clustering through deep generative models.
For example, Variational Deep Embedding (VaDE) (Jiang et al., 2017) proposed a clustering framework combining VAE and GMM together.
Also, Gaussian Mixture Variational Autoencoder (GMVAE)
(Dilokthanakul et al., 2016) built upon the semi-supervised model by Kingma et al.
(2014) to perform unsupervised clustering within the VAE framework with a Gaussian mixture as a prior distribution.
GAN-based methods include: Categorical Generative Adversarial Networks (CatGAN) (Springenberg, 2015), an approach incorporating neural network classiﬁers with an adversarial generative model, and Adversarial Autoencoder (AAE) (Makhzani et al., 2015), a probabilistic autoencoder variant integrating traditional reconstruction error with adversarial training criterion of
GANs.
Besides, Premachandran & Yuille (2016) proposes to fuse the disentangled features learned by Information Maximizing Generative Adversarial Networks (InfoGAN), an extension to GANs that uses mutual information to induce representation, with k -means clustering.
In this paper, we propose a novel unsupervised clustering approach building upon the previous study on learning of latent annotations in a particular semi-supervised setting where a coarse level of supervision is available for all observations, i.e. parent-class labels, but the model has to learn a ﬁne level of latent annotations, i.e. sub-classes, under each one of these parents.
For clariﬁcation, assume that we are given a dataset of hand-written digits such as MNIST (LeCun et al., 1998) where the overall task is the complete categorization of each digit, but the only available supervision is whether a digit is smaller or greater than 5.
To study this particular semi-supervised setting on neural networks, Kilinc & Uysal (2017a) proposed a novel output layer modiﬁcation, Auto-clustering
Output Layer (ACOL).
ACOL allows simultaneous supervised classiﬁcation (per provided parentclasses) and unsupervised clustering (within each parent) where clustering is performed through
Graph-based Activity Regularization (GAR) technique recently proposed in Kilinc & Uysal (2017b).
More speciﬁcally, as ACOL duplicates the softmax nodes at the output layer for each class, GAR allows for competitive learning between these duplicates on a traditional error-correction learning framework.
To learn latent annotations in a fully unsupervised setup, we substitute the real, yet unavailable, parentclass information with a pseudo one.
More speciﬁcally, we choose a domain speciﬁc transformation to be applied to the observations in a dataset to generate examples for a pseudo parent-class.
The transformed dataset constitutes the examples of that pseudo parent-class and every new transformation generates a new one.
Regarding the MNIST example for this fully unsupervised setting, now we simply augment the dataset by applying a transformation to examples, e.g. rotating by 90 o , and label transformed examples as rotated and non-transformed examples as original .
This new augmented dataset is provided to the network as a two-class classiﬁcation problem with pseudo classes labeled as original and rotated as visualized in Figure 1.
While being trained over this pseudo supervision, through ACOL and GAR, the neural network learns the latent representation distinguishing the real digit identities in an unsupervised manner.
The idea of employing an auxiliary task to learn a good data representation has been previously studied for different domains (Collobert et al., 2011; Ahmed et al., 2008).
Most recent study, Exemplar CNN (Dosovitskiy et al., 2016), proposed to use a regularizer enforcing the feature representation to be approximately invariant to the transformations while training the network to discriminate between a set of pseudo parent-classes (“surrogate classes” with their deﬁnition).
This approach requires thousands of transformations to obtain a good representation and also it cannot exploit more than 300 examples per “surrogate class” severely limiting its scalability.
Furthermore, some elementary
[Figure]
rotation
B ACKGROUND 2. A UTO - CLUSTERING O UTPUT L AYER
Unlike traditional output layer structure, the Auto-clustering Output Layer (ACOL) (Kilinc & Uysal, 2017a) deﬁnes more than one softmax node ( k s duplicates) per parent-class.
Outputs of k s duplicated softmax nodes that belong to the same parent are then combined in a subsequent pooling layer for the ﬁnal prediction.
Training is performed in the conﬁguration shown in Figure 2 where n p is the number of parent-classes.
This might look like a classiﬁer with redundant softmax nodes.
However, duplicated softmax nodes of each parent are specialized using GAR throughout the training in a way that each one of n = n p k s softmax nodes represent an individual sub-class of a parent, i.e. annotation.
In order to mathematically describe this modiﬁcation, let us consider a neural network with L −hidden layers where l denotes the individual index for each layer such that l ∈ { 0 , ..., L } .
Let Y ( l ) denote the output of the nodes at layer l .
Y (0) =
X is the input and f ( X ) =
f ( L ) ( X ) =
Y ( L ) =
Y is the output of the entire network.
W ( l ) and b ( l ) are the weights and biases of layer l , respectively.
Then, the feedforward operation of the neural networks can be written as
[Equation]
where h ( l ) (.) is the activation function applied at layer l .
For ACOL networks, h ( L − 1) ( . ) and h ( L ) ( . ) respectively correspond to softmax and linear activation functions.
Also, W ( L ) :=
[ I n p . . .
I n p ] T and b ( L ) := 0 where I denotes the identity matrix as
ACOL simply deﬁnes constant weights between the augmented softmax layer and the pooling layer to sum up the output probabilities of the softmax nodes belonging to the same parent.
Let Z denote the activities at the input of augmented softmax layer such that
[Equation]
[Figure]
corresponding to an m × n matrix where m is the number of examples and n is the total number of all softmax nodes at the augmented softmax layer such that n = n p k s , where n p is the number of parent-classes and k s is the clustering coefﬁcient of ACOL.
Then, the output of the ACOL applied network can be written in terms of Z as
[Equation]
2. G RAPH - BASED A CTIVITY R EGULARIZATION
Kilinc & Uysal (2017a) adopted the Graph-based Activity Regularization (GAR) technique (Kilinc &
Uysal, 2017b) as the unsupervised regularization term to create competition between the duplicated softmax nodes of the augmented softmax layer which ultimately results in specialized but equallyactive softmax nodes each representing a latent annotation within a parent.
The GAR technique applies the regularization over the positive part of the activities at the input of softmax nodes such that
[Equation]
and deﬁnes two terms to turn n × n symmetric matrix N , which is deﬁned as N := B T B , into the identity matrix.
While the afﬁnity term penalizes the non-zero off-diagonal entries of N , balance attempts to equalize diagonal entries.
Let v be a 1 × n vector representing the diagonal entries of
N such that v :=
[ N 11 . . .
N nn ] and V be deﬁned as n × n symmetric matrix such that V := v T v .
Then, the afﬁnity and balance terms can be written as
[Equation]
which modiﬁes the overall objective function of the training proposed in Kilinc & Uysal (2017a) as
[Equation]
where L ( . ) is the supervised log loss function, t
=
[ t 1 . . .
t m ] T is the vector of provided parent-class labels such that t i ∈ { 1 , ..., n p } (recall that, in the semi-supervised setting considered in Kilinc &
Uysal (2017a)
, there is a real partial supervision available for all examples, e.g. a digit is smaller or greater than 5), U ( . ) is the unsupervised regularization term consisting of afﬁnity , balance and || B || F (the Frobenius norm for B ) that is employed to limit the denominators of both afﬁnity and balance terms not to diminish their effects and c α , c β , c F are the weighting coefﬁcients.
GAR has been originally proposed for the classical type of semi-supervised setting where the number of labeled observations is much smaller than the number of unlabeled observations, but all existing classes are equally represented by the available labels even at limited numbers.
Kilinc & Uysal (2017b) have shown that deﬁning the objective of the regularization over the matrix N yields a scalable and efﬁcient graph-based solution and that the entire operation corresponds to propagating the available labels across the graph G M whose edges are speciﬁed by the m × m symmetric matrix
M := BB T that infers the adjacency of the examples based on the predictions of the neural network.
More speciﬁcally, it has been shown that as the matrix N turns into the identity matrix, G M becomes a disconnected graph including n disjoint subgraphs each of which is m / n -regular.
This indicates that the strong adjacencies in the matrix M get stronger, weak ones diminish and each label is propagated to m / n examples through the strong adjacencies.
On the other hand, in the particular semi-supervised setting considered by Kilinc & Uysal (2017a) (i.e. a coarse level of labeling is available for all observations but the model still needs to learn a ﬁne level of latent annotation for each one of them), when applied to an ACOL network, GAR provides that the latent information introduced by the coarse supervision is propagated from the graph G Y (whose edges are speciﬁed by m × m symmetric matrix Y Y T ) to its spanning subgraph G M to reveal deeper latent annotations.
In other words, although these two graphs are made up of the same vertices ( m examples) while propagating the latent information that is captured through supervised adjacency introduced by G Y across G M , GAR terms eliminate some of the edges of G Y from G M in a way that G M ultimately becomes a disconnected graph of n disjoint subgraphs each of which now corresponds to a latent annotation.
P ROPOSED F RAMEWORK 3. O BJECTIVE F UNCTION
The unsupervised clustering approach proposed in this paper adopts the same framework introduced in Kilinc & Uysal (2017a).
Since the real parent-class labels (a digit is smaller or greater than 5) are unavailable in a fully unsupervised setting, we randomly assign pseudo parent-class labels each of which is associated with a domain speciﬁc transformation used to generate the examples of that pseudo parent-class.
In this setting, n p now corresponds to the number of pseudo parent-classes and ˜ t =
[˜ t 1 . . .
˜
t m ] T is a vector of randomly assigned pseudo parent-class labels which are uniformly distributed across n p pseudo parent-classes such that ˜ t i ∈ { 1 , ..., n p } .
Also, there exists a set of transformations
S T =
{T 1 , ..., T n p } where transformation T j is used to generate the examples of the j th pseudo parentclass such that ˜ x
i = T j ( x i ) .
S T also includes non-transformation T 1 providing ˜ x
i =
T 1 ( x i )
=
x i to ensure that the original observations are introduced to the network during training.
˜ t is associated with a vector of transformations T =
[ T 1 . . .
T m ] T such that T
i = T ˜ t i .
Let (cid:12) be an element-wise operation deﬁned between the vector of transformations T and the original input X
=
[ x 1 . . .
x m ] T such that
[Equation]
where ˜ X corresponds to the modiﬁed input per randomly assigned pseudo labels ˜ t .
The output of the entire network and the positive part of the augmented softmax layer activities respectively become
Y = f
( ˜ X ) and B = g ( ˜ X ) .
Then, the objective function deﬁned in (7) can simply be adopted by substituting the real, yet unavailable, observation-label pair ( X , t ) with a pseudo one ( ˜ X , ˜ t ) such that
[Equation]
3. M ODIFIED Afﬁnity AND Balance T ERMS
Recall that an n × n symmetric matrix N = B T B speciﬁes the edges of the graph between the softmax duplicates and that GAR terms have been proposed to regularize the matrix N in a way that it turns into the identity matrix.
While the objective of afﬁnity , i.e. penalizing the non-zero off-diagonal entries of N , corresponds to assigning an example to only one softmax node with the probability of 1, the objective of balance , i.e. equalizing diagonal entries of N , corresponds to preventing collapsing onto a subspace of dimension less than n .
Among the off-diagonal entries of N determining the afﬁnity cost, for each one of n softmax nodes, there exist k s − 1 entries describing its relation with the other duplicates of the same parent-class (let us deﬁne them as intra-parent entries) and ( n p − 1) k s entries describing its relation with the softmax nodes belonging to other parent-classes (let us deﬁne them as inter-parent entries).
While inter-parent entries are explicitly affected by the pseudo classiﬁcation objective as well as the regularization, intra-parent entries do not experience the classiﬁcation directly.
Therefore, the afﬁnity cost due to inter-parent entries is minimized at a different rate than the afﬁnity cost due to intra-parent entries.
On the other hand, as it is calculated over the diagonal entries of N , the balance cost does not either experience the pseudo classiﬁcation objective explicitly.
As a result, due to the direct impact of the pseudo classiﬁcation objective which is observed only on the afﬁnity cost, the weighting between the regularization terms actively alters during the training and needs to be re-tuned through the hyperparameters c α and c β .
This effect can be observed more clearly as n p , the number of parent-classes, increases.
To ensure a more robust regularization we introduce a modiﬁcation for the afﬁnity and balance terms:
We discard all inter-parent entries of N and represent the remaining ones as a three dimensional tensor ˜ N .
Thus, ˜ N is a k s × k s × n p tensor such that ˜ N : , : ,k speciﬁes the relations between k s softmax duplicates of the k th parent-class where k ∈ { 1 , ..., n p } .
Also, ˜ V is another k s × k s × n p tensor deﬁned as
[Equation]
Then, the modiﬁed afﬁnity and balance terms can be respectively written as
[Equation]
[Equation]
and simply correspond to calculating the original terms given in (5), (6) on each 2-D k s × k s ×slice of ˜ N and ˜ V tensors and then averaging the results for n p of them.
Replacing these modiﬁed terms in (9), the overall modiﬁed objective function becomes
[Equation]
3. T RAINING AND C LUSTER A SSIGNMENTS
Network parameters are trained by implementing the stochastic optimization method Adam (Kingma & Ba, 2014) based on the objective given in (13).
After training, k -means clustering is performed on the representation space observed in the hidden layer preceding the augmented softmax layer such that
[Equation]
Recalling that the original examples are already introduced to the network as the examples of ﬁrst pseudo parent-class through transformation T 1 , we obtain the latent space representation only for the original examples to perform k -means clustering.
One might suggest performing k -means clustering on the representation observed in the augmented softmax layer ( Z or softmax( Z ) ) rather than F .
Properties and respective clustering performances of these representation spaces are empirically demonstrated in the following sections.
Algorithm 1 below describes the entire training and cluster assignment procedure.
Input : X = [ x 1 . . .
x m ] T , n p , a set of transformations S T =
{T 1 , ..., T n p } , batch size b , weighing coefﬁcients c α , c β , c F , the number of clusters k repeat ˜ t
←− random ( n p ) //
Randomly assign labels across n p classes
T ←− [ T ˜ t 1 , ..., T ˜ t m ] // Obtain the vector of transformations corresponding to ˜ t ˜ X ←− T (cid:12) X //
Obtain the modified input (cid:8) (
X 1 , ´ t 1 ) , ..., ( ´ X m / b , ´ t m / b ) (cid:9) ←− X , create pairs i do Forward
| || met
Obtain latent space representation F for the original examples y kmeans
Assign clusters by performing k means on F return: y
E XPERIMENTS 4. E XPERIMENTAL S ETUP AND D ATASETS
The models have been implemented in Python using Keras (Chollet, 2015) and Theano (Theano
Development Team, 2016).
Open source code is available at http://github.com/ozcell/LALNets that can be used to reproduce the experimental results obtained on three benchmark image datasets, MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011) and USPS.
Speciﬁcations of these datasets are presented in Table 1.
[Table]
All experiments have been performed on a 6-layer convolutional neural network (CNN) model whose speciﬁcations are given in Table 2 where coefﬁcients of GAR terms have been chosen as k s = 20 , c α = 0 .
1 , c β = 1 , c F = 0 .
During training, pseudo supervised objective is introduced as an 8 pseudo parent-class classiﬁcation problem, i.e. n p = 8 , through the following rotation-based transformations:
[Equation]
= 1 : No transformation i = 2 : Rotate by 90 o i = 3 : Rotate by 180 o i = 4 : Rotate by 270 o i = 5 :
Flip horizontally i = 6 : Flip horizontally + Rotate by 90 o i = 7 : Flip horizontally + Rotate by 180 o i = 8 : Flip horizontally + Rotate by 270 o (15)
For all experiments, we used a batch size of 400 and each experiment has been repeated 10 times.
To ensure that the representation obtained through the proposed approach is well-generalized for never-seen-before data, we train the neural network parameters using only the training set examples of each dataset and obtain the clustering performances using k -means with k = 10 on the latent space representation F of the untransformed test set examples (through T 1 ).
[Table]
4. Q UANTITATIVE C OMPARISON
Following Jiang et al.
(2017) and Yang et al.
(2016), we evaluate the test performances using unsupervised clustering accuracy given as
[Equation]
where t ∗ i is the ground-truth label, y i is the assigned cluster, and F is the set of all possible one-to-one mappings between assignments and labels.
Both metrics range between [0 , 1] where a larger value indicates more precise clustering results.
Figure 3 presents the t-SNE (Maaten & Hinton, 2008) visualizations of the latent space F throughout the training for 2000 untransformed test examples from MNIST.
Each group corresponds to a cluster (i.e. a digit) under the ﬁrst pseudo parent-class (i.e. the class of untransformed examples including all ten digits).
Color codes denote the ground-truths for the digits.
From epoch 1 to epoch 400 of the unsupervised (but pseudo supervised) training, clusters become well-separated and simultaneously the clustering accuracy increases.
As clearly observed from this ﬁgure, using the pseudo supervision, the neural network also reveals some hidden patterns useful to distinguish the real digit identities and ultimately learns to categorize each one of them.
It is also worth noting that a high level of clustering accuracy is achieved relatively quickly (after only 50 epochs) as seen both in the t-SNE and test accuracy plots.
[Table]
[Figure]
Table 3 summarizes quantitative unsupervised clustering performances observed on three datasets in terms of unsupervised clustering accuracy (ACC).
Results of a broad range of recent existing solu- tions are also presented for comparison.
These solutions are grouped according to their approaches to unsupervised clustering.
Following the very recent developments in deep generative models,
VaDE (Jiang et al., 2017) and GMVAE (Dilokthanakul et al., 2016) employ variational autoencoders while CatGAN (Springenberg, 2015), AAE (Makhzani et al., 2015) and IMSAT (Hu et al., 2017) adopt adversarial training.
DEC (Xie et al., 2016) simultaneously learns feature representations and cluster assignments using DNNs.
On the other hand, JULE (Yang et al., 2016) combines agglomerative clustering with CNNs.
Also, the performances of two conventional approaches, applying k -means on raw data space and applying k -means on the autoencoder representation, are provided to show a baseline for unsupervised clustering performances.
Our approach statistically signiﬁcantly outperforms all the contemporary methods that reported unsupervised clustering performance on
MNIST except IMSAT (Hu et al., 2017) displaying very competitive performance with our approach, i.e. 98 .
32%( ± 0 . 08) vs. 98 .
40%( ± 0 . 40) .
However, results obtained on the SVHN dataset, i.e.
80%( ± 1 . 30) vs. 57 .
30%( ± 3 . 90) , show that our approach statistically signiﬁcantly outperforms
IMSAT on this realistic dataset and deﬁnes the current state-of-the-art for unsupervised clustering on
SVHN.
Besides, the USPS dataset provides another basis of comparison between our approach and
JULE.
[Table]
Jiang et al
[Table] [Table]
GMVAE (Dilokthanakul et al., 2016)
[Table]
GMVAE (Dilokthanakul et al., 2016)
[Table]
GMVAE (Dilokthanakul et al., 2016)
[Table] [Table]
(Springenberg, 2015)
[Table] [Table]
(Makhzani et al., 2015)
[Table] [Table]
(Makhzani et al.
[Table]
[Table] [Table] [Table]
[Table]
k
[Table]
means (Xie et al., 2016)
[Table]
AE+ k -means (Xie et al., 2016)
[Table]
DEC (Xie et al., 2016)
[Table]
[Table] JULE (Yang [Table] al., 2016)
[Table]
Our approach
[Table]
4. R EPRESENTATION P ROPERTIES
Recall that, for the 6-layer CNN model employed in the experiments, F = Y ( L − 2) corresponds to the output of the fully-connected layer of 2048 ReLU nodes, Z = F W L − 1 + b L − 1 is the input of the augmented softmax layer of 160 nodes, i.e. n = n p k s , where 8 pseudo parent-classes are represented by 20 softmax duplicates each.
Figure 4 provides the average value for each dimension of F , Z and softmax( Z ) observed with respect to untransformed test set examples and the norm of the associated weights.
Note that the representation on F is not distributed to the entire space but the weights associated to these unused dimensions do not decay.
On the other hand, due to the pseudo supervision task, the output of the augmented softmax layer i.e. softmax( Z ) , becomes a one-hot encoded representation of whichdimensions, i.e. ( n p − 1) k s , are inactive for the untransformed examples; however, the representation at its input is distributed to all dimensions.
Figure 4 also summarizes how the dimension size of
F , i.e. the number of ReLU nodes in the fully-connected layer, affects the clustering performance.
Decreasing the number of dimensions of F up to a point, i.e. ≈ 1024 , does not signiﬁcantly affect the clustering accuracy.
However, further decrease beyond this point dramatically reduces the performance.
[Figure]
For comparison, Figure 5 presents t-SNE visualizations of these latent representations observed with respect to 2000 untransformed test examples from MNIST.
One can clearly see that clusters are not well-separated on one-hot encoded softmax( Z ) ; however, separations of the clusters are quite similar and clear on the representation spaces F and Z .
Hence, one can also obtain similar clustering accuracy, i.e. = 98 .
16% ± (0 . 14) , by applying k -means on the representation space Z .
[Figure]
4. G RAPH I NTERPRETATION OF THE L ATENT I NFORMATION P ROPAGATION THROUGH GAR
Recall that GAR terms have been originally proposed to propagate the available labels towards the unlabeled examples in a semi-supervised setting and Kilinc & Uysal (2017a) have shown that these terms can also be adopted to propagate the hidden information that is introduced by a coarse level of supervision and which is useful to discover a deeper level of latent annotations.
In the fully unsupervised setting considered in this paper, as no real supervision is available, hidden information useful to discover unknown clusters is now captured through the help of domain speciﬁc transformations and propagated by GAR terms as well.
Figure 6 visualizes the realization of this propagation using the real predictions obtained on MNIST.
Colored circles denote the ground-truths for the vertices, i.e. examples, and gray lines denote the edges, i.e. non-zero weighted connections between the examples representing their similarity.
Note that, for vertices in graph G Y , there are two different colors indicating true pseudo parent-class labels assigned per the applied transformation (for simplicity, out of 8, only the examples of the ﬁrst two pseudo parent-classes are used for this illustration), albeit ten different colors indicating the real digit identity for vertices in graph G M .
Recall that edges of these two graphs, E Y and E , are respectively inferred by matrices Y Y T and BB T where B = max(0 , Z ) and that G M is the spanning subgraph of G Y .
That is, G M =
( M , E ) shares the same vertices M with graph G Y =
( M , E Y ) , which is constructed per the pseudo supervision; however, E is a subset of E Y as some of the edges in graph
G Y , such as those between the examples of digit 0 and 1, are eliminated in graph G M due to GAR regularization terms.
As training continues, pseudo supervision eliminates the edges between the examples of different pseudo parent-classes and turns graph G Y into a disconnected graph of n p =disjoint subgraphs (only two of them are illustrated).
Simultaneously, GAR terms eliminate the edges between the examples of the same parent-class in graph G M to discover previously unknown clusters.
Ultimately, G M becomes disconnected graphs of δ disjoint subgraphs where n p ≤ δ ≤ n p k s and each disjoint subgraph corresponds to a cluster.
[Figure]
only for the simplicity of the visualization).
Colored circles denote the ground-truths for the vertices, i.e. examples, and gray lines denote the edges, i.e. non-zero weighted connections between the examples representing their similarity.
Note that, for vertices in graph G Y , there are two different colors indicating true pseudo parent-class labels assigned according to the applied transformation (for simplicity, out of 8, only the examples of ﬁrst two pseudo parent-classes are used for this illustration), albeit ten different colors indicating the real digit identity for vertices in graph G M .
As training continues, pseudo supervision eliminates the edges between the examples of different pseudo parent-classes and turns graph G Y into a disconnected graph of n p = 8 disjoint subgraphs (only two of them are illustrated).
Simultaneously, GAR terms eliminate the edges between the examples of the same parent-class in graph G M to discover previously unknown clusters.
Ultimately, G M becomes disconnected graphs of δ disjoint subgraphs where n p ≤ δ ≤ n p k s and each disjoint subgraph corresponds to a cluster.
This ﬁgure is best viewed in color.
4. T HE I MPACT OF THE N UMBER OF C LUSTERS k
For the quantitative clustering results, we set the number of clusters for the k -means to the number of classes assuming a prior knowledge, i.e. k = 10 .
To demonstrate the representation power of the proposed approach as an unsupervised clustering model, on MNIST, we deliberately choose different k values for the k -means clustering applied on the representation space F .
For two different k settings i.e. 7 and 20, Figure 7 illustrates a few examples of each cluster.
One can see that when k is smaller than the actual number of classes, digits with similar appearances are grouped together, such as digits 4 and 9, 5 and 8, 0 and 6.
When k is set to a bigger value than the number of classes, some digits are divided into subclasses based on visually identiﬁable image properties such as digit tilt, roundness, etc.
Note the differences between upright and oblique digit 1 as shown in clustersand 20, between two styles of digit 6 as shown in clusters 18 and 19, and between two styles of digit 2 as shown in clusters 7 and 12.
[Figure]
When k is smaller than the actual number of classes, digits with similar appearances are grouped together, such as digits 4 and 9, 5 and 8, 0 and 6.
When k is set to a bigger value than the number of classes, some digits are divided into subclasses based on visually identiﬁable image properties such as digit tilt, roundness, etc.
Note the differences between upright and oblique digit 1 as shown in clusters 2 and 20, between two styles of digit 6 as shown in clusters 18 and 19, and between two styles of digit 2 as shown in clusters 7 and 12.
4. T HE I MPACT OF T RANSFORMATIONS
As the revealed unknown clusters are directly related with the captured latent information through pseudo parent-classes, choosing the right set of transformations for the clustering task of concern is crucial for the performance.
Figure 8 presents t-SNE visualizations of the representation spaces observed when different sets of transformations are adopted.
The ﬁrst row of Figure 8 illustrates the clustering results when one of four different transformation types, i.e. scaling, shearing, translation and random permutation of the pixels, is applied variably to generate 8 pseudo parent-classes.
One can observe some level of grouping with scaling and shearing-based transformations; however, the clusters deﬁned by these groupings do not represent real digit identities (as shown by the colored dots) and may indicate other features of images.
On the other hand, translating the images or randomly permuting the pixel positions do not provide any useful knowledge to discover any well-deﬁned clustering.
The second row of Figure 8 presents the results obtained when rotation-based transformations listed in (15) are adopted.
One can easily observe that only two or four pseudo parent-classes generated using rotation-based transformations are sufﬁcient to obtain decent clustering representing the real digit identities.
Considering that, for MNIST, the clustering accuracy obtained using all 8 transformations in (15) is 98 .
32%( ± 0 . 08) , we have achieved 97 .
80%( ± 0 .
18) accuracy using S T
= {T 1 , T 2 , T 3 , T 4 } , 72 .
52%( ± 6 . 20) accuracy using S T
= {T 1 , T 2 } and 96 .
84%( ± 0 . 29) accuracy using S T
= {T 1 , T 3 } .
Recalling that T 2 and T 3 respectively correspond to rotating the images by 90 o and 180 o , one can say that comparing the untransformed images with their 180 o rotated versions is more effective in terms of capturing the latent information that is useful to distinguish the real digit identities.
In fact, T 3 alone is sufﬁcient to achieve state-of-the-art clustering accuracy on MNIST.
Adding more rotation-based transformations to S T further improves the clustering performance.
To summarize, the type of the transformation generating the pseudo parent-classes is more important than their number and different transformations can reveal different clustering patterns.
Therefore, ﬁnding the right transformation type for the clustering task of concern is crucial for the proposed approach in this paper and it remains an important research question how to identify the kind of transformation most optimized for the clustering task at hand.
[Figure]
C ONCLUSION
In this paper, we introduced a novel unsupervised clustering approach building upon the previous study on an output layer modiﬁcation, ACOL, which is proposed to learn latent annotations on neural networks when a partial supervision is provided.
To discover unknown clusters in a fully unsupervised setup, we substitute the real, yet unavailable, partial supervision with a pseudo one.
More speciﬁcally, we randomly assign pseudo parent-class labels each of which is associated with a different domain speciﬁc transformation.
Each observation is modiﬁed by applying the transformation corresponding to the assigned pseudo label.
Generated observation-label pairs are used to train an
ACOL network that introduces multiple softmax nodes for each pseudo parent-class.
Due to the unsupervised regularization based on GAR terms, each softmax duplicate under a parent-class is specialized as the latent information captured by the help of domain speciﬁc transformations is propagated throughout the training.
Ultimately we obtain a k -means friendly latent representation.
Furthermore, we demonstrate that the neural network can learn by comparing differently transformed examples and translate that knowledge to reveal unknown clusters.
The proposed approach was validated on three image benchmark datasets, MNIST, SVHN and USPS, through t-SNE visualizations and unsupervised clustering accuracy exceeds those reported by well-accepted approaches in the literature.
Future work will extend this approach to other domains such as sequential data.
We will also explore how to optimize domain speciﬁc transformations based on known or otherwise identiﬁable characteristics of the dataset being considered for clustering.