G-CEALS: G AUSSIAN C LUSTER E MBEDDING IN A UTOENCODER L ATENT S PACE FOR T ABULAR D ATA R EPRESENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The latent space of autoencoders has been improved for clustering image data by jointly learning a t distributed embedding with a clustering algorithm inspired by the neighborhood embedding concept proposed for data visualization However , multivariate tabular data pose different challenges in representation learning than image data, where traditional machine learning is often superior to deep tabular data learning.
In this paper, we address the challenges of learning tabular data in contrast to image data and present a novel Gaussian Cluster Embedding in Autoencoder Latent Space (G-CEALS) algorithm by replacing t-distributions with multivariate Gaussian clusters.
Unlike current methods, the proposed method defines the Gaussian embedding and the target cluster distribution independently to accommodate any clustering algorithm in representation learning.
A trained GCEALS model is used to extract a quality embedding for unseen test data. Based on the embedding clustering accuracy, the average rank of the proposed G-CEALS method is which is superior to all eight baseline clustering and cluster embedding methods on seven tabular data sets.
This paper shows one of the first algorithms to jointly learn embedding and clustering for improving the representation of multivariate tabular data in downstream clustering
I NTRODUCTION
Deep learning has replaced traditional machine learning in many data-intensive research and applications due to its ability to perform concurrent and efficient representation learning and classification.
This concurrent learning approach outperforms traditional machine learning that requires handcrafted features to perform supervised classification (Lin et al., 2020; Alam et al., 2020).
However, representation learning via supervisory signals from ground truth labels may be prone to overfitting Ying (2019) and adversarial attacks (Huang et al., 2017).
Moreover, human annotations for supervised representation learning and classification may not be available in all data domains or for all data samples.
To address these pitfalls, representation learning via unsupervised clustering algorithms may be a strong alternative to supervised learning methods.
The limitation of supervised representation learning may be overcome using self-supervision or pseudo labels that do not require human-annotated supervisory signals (Boubekki et al., 2021; Caron et al., 2018).
In a self-supervised autoencoder, the objective is to preserve all information of input data in a low-dimensional embedding for data reconstruction.
However, embeddings for data reconstruction do not emphasize representations essential for downstream classification or clustering tasks.
Therefore, unsupervised methods have been proposed for jointly learning embedding with clustering to yield clustering friendly representations (Xie et al., 2016;
Guo et al., 2017; Moradi
Fard et al., 2020; Mrabah et al., 2020; Yang et al., 2017).
The existing cluster embedding literature suggests several strict assumptions about clustering algorithms (k-means), cluster distributions (t-distribution), and data modality (image data).
While deep representation learning of image data is well studied using convolutional neural networks (CNN), deep learning has not seen much success with structured tabular data.
There is strong evidence in the literature that traditional machine learning still outperforms deep models in learning tabular data (Kohler et al., 2019; Smith et al., 2020; Borisov et al., 2021; Kadra et al., 2021; Shwartz-Ziv & Armon, 2022).
In this paper, we review the assumptions made in the cluster embedding literature and revise those assumptions for the representation learning of tabular data.
Accordingly, a novel joint learning framework is proposed considering the architectural and algorithmic differences in learning image and tabular data.
The remainder of this manuscript is organized as follows.
Section 2 provides a review of the state-ofthe-art literature on deep cluster embedding.
Section 3 introduces tabular data with some theoretical underpinnings of neighborhood embedding and cluster embedding in support of our proposed representation learning framework.
Section 4 outlines the proposed joint cluster embedding framework to obtain a quality representation of tabular data for downstream clustering or classification.
Section 5 summarizes the tabular data sets and experiments for evaluating the proposed joint learning framework.
Section 6 provides the results following the experiments and compares our proposed method with similar methods in the literature.
Section 7 summarizes the findings with additional insights into the results and limitations.
The paper concludes in Section 8.
R ELATED WORK
One of the earliest studies on cluster embedding, Deep Embedded Clustering (DEC
) Xie et al.
(2016), is inspired by the seminal work on t-distributed stochastic neighborhood embedding (tSNE) (Van Der Maaten & Hinton, 2008).
The DEC approach first trains a deep autoencoder by minimizing the data reconstruction loss.
The trained encoder part (excluding the decoder) is then fine-tuned by minimizing the Kullback-Leibler (KL) divergence between a t-distributed cluster distribution (Q) on the embedding and a target distribution (P).
The target distribution is obtained via a closed-form solution by taking the first derivative of the KL divergence loss between P and Q distributions with respect to P and equating it to zero.
Therefore, the assumption of t-distribution holds for both Q and P distributions in similar work.
The k-means clustering in the DEC approach is later replaced by spectral clustering to improve the quality of embedding in terms of clustering performance (Duan et al., 2019).
The DEC approach is also enhanced by an improved DEC (IDEC) framework (Guo et al., 2017).
In IDEC, the autoencoder reconstruction loss and the KL divergence loss are jointly minimized to update the weights of a deep autoencoder and produce the embedding.
Similar approaches, including t-distributions, k-means clustering, and KL divergence loss, are adopted in joint embedding and cluster learning (JECL) for multimodal representation learning of text-image data pairs (Yang et al., 2020).
The Deep Clustering via Joint Convolutional Autoencoder (DEPICT) approach learns image embedding via a de-noising autoencoder (Dizaji et al., 2017).
The embedding is mapped to a softmax function to obtain a cluster distribution or likelihood (Q) instead of assuming a distribution.
Following a series of mathematical derivations and assumptions, their final learning objective includes a cross-entropy loss involving P and Q distributions and an embedding reconstruction loss for each layer of the convolutional autoencoder.
A general trend in the cluster embedding literature shows that K-means is the most common clustering method (Xie et al., 2016;
Guo et al., 2017; Mrabah et al., 2020; Moradi Fard et al., 2020; Zhang et al., 2019;
Yang et al., 2020; Dizaji et al., 2017).
The assumption of t-distributed cluster embedding made in the DEC method Xie et al.
(2016) continues to appear in the literature Ren et al.
Enguehard et al.
; Guo et al.
(2017); Duan et al.
; Yang et al.
; Wu et al. without any alternatives.
The assumption of t-distribution is originally made in the t-SNE algorithm for data visualization using neighborhood embedding maps (Van Der Maaten & Hinton, 2008).
We argue that the assumptions of neighborhood embedding for data visualization are not aligned with the requirements of cluster embedding.
Moreover, cluster embedding methods proposed in the literature are invariably evaluated on benchmark image data sets.
The methods for image learning may not be optimal or even ready to learn tabular data representations.
To the best of our knowledge, similar cluster embedding methods have not been studied on multivariate tabular data.
2. C ONTRIBUTIONS
This paper is one of the first to investigate the performance of joint cluster embedding methods on tabular data. The limitations of state-of-the-art joint cluster embedding methods are addressed to contribute a new cluster embedding algorithm as follows.
First, we replace the current assumption of t-distributed embedding with a mixture of multivariate Gaussian distributions for multivariate tabular data by providing a theoretical underpinning for this choice.
Second, a new cluster embedding algorithm is proposed using multivariate Gaussian distributions that can jointly learn distributions with any clustering algorithm.
Third, we define the target cluster distribution on the tabular data space instead of deriving it from the embedding because traditional machine learning of tabular data is still superior to deep learning and can add complementary benefits to the embedding learned via an autoencoder.
Therefore, our embedding and target distributions are independent of each other to flexibly learn any target cluster distribution depending on the application domain.
T HEORETICAL BACKGROUND
This section provides preliminaries on tabular data in contrast to image data. We draw multiple contrasts between neighborhood embedding proposed for data visualization and cluster embedding proposed for representation learning to underpin our proposed approach.
3. P RELIMINARIES
A tabular data set is represented in a matrix X ∈ ℜ n × d with n i.i.d samples in rows.
Each sample ( X i ) is represented by a d-dimensional feature vector, X i ∈ ℜ d = { x 1 , x 2 , ..., x d } , where i = { 1 , 2 , ..., n } .
Compared to a pixel distribution P ( I ) of an image I , tabular data contain multivariate distributions P ( x 1 , x 2 , ..., x d ) of heterogeneous variables often in relatively lower dimensions.
Although sequential (genomics) or time-series data can be high-dimensional and structured in data tables, those require special learning architectures.
In general, tabular data in business, health records, and many domains fail to take advantage of deep convolutional learning due to the absence of sequential patterns or image-like spatial regularities.
Therefore, tabular data sets are identified as the last ”unconquered castle” for deep learning Kadra et al.
(2021), where traditional machine learning methods are still competing strongly against advanced neural network architectures Kadra et al.
; Borisov et al.
Unlike image learning, there is a need for robust tabular data learning methods to outperform the superior traditional machine learning or clustering methods.
A neighborhood embedding is a low-dimensional map that preserves the similarity between data points ( x i and x j ) observed in a higher dimension.
Maaten and Hinton propose a Student’s tdistribution to model the similarity between samples in neighborhood embedding ( z i , z j ) of highdimensional data points ( x i and x j ) for data visualization (Van Der Maaten & Hinton, 2008).
First, the similarity between two sample points ( x i and x j ) in the high dimension is modeled by a Gaussian distribution, p ij in Equation 1.
Similar joint distribution can be defined for a pair of points in the low-dimensional embedding ( z i , z j ) as q ij below.
[Equation]
The divergence between the target ( p ij ) and embedding ( q ij ) distributions is measured using a KL divergence loss, which is minimized to iteratively optimize the neighborhood embedding.
[Equation]
To facilitate high-dimensional data visualization in two dimensions (2D), the embedding distribution ( q ij ) is modeled by a Student’s t-distribution, as shown in Equation 3.
One primary justification for tdistribution is its heavier tails compared to a Gaussian distribution.
A heavier tail aids in an efficient mapping of outliers observed in high dimensional space to the 2D space for data visualization.
[Equation]
Therefore, data points placed at a moderate distance in high-dimension are pulled farther by a tdistribution to aid visualization in 2D space.
In the context of cluster embedding, we argue that the additional separation between points in low dimensions may alter their cluster assignments.
To illustrate this phenomenon, we project high-dimensional deep convolutional image features on 2D using 1) t-SNE and 2) two principal components, as shown in Figure 2 (Appendix A).
The scattering of data points is evident in the t-SNE mapping, where one blue point appears on the left side of the figure leading to a wrong cluster assignment, unlike the PCA mapping (Figure 2 (b) in Appendix A).
In general, the expectations of data visualization and clustering tasks are different, as highlighted in
Table 4 (Appendix A), which should be considered in respective representation learning.
3. C LUSTER EMBEDDING
Cluster embedding is achieved by infusing cluster separation information into the low-dimensional latent space.
While neighborhood embedding is initialized by sampling from a Gaussian distribution, cluster embedding methods use embedding learned from an autoencoder’s latent space.
However, the current cluster embedding methods use the same t-distribution (Equation 3) to define the embedding distribution ( q ij ), similar to neighborhood embedding.
The target distribution ( p ij ) is derived as a function of q ij , as shown below.
[Equation]
While pair-wise sample distances in neighborhood embedding have a complexity of O ( N 2 ), the distances from the centroids in embedding are O(N*K).
Here, K is the number of clusters, which is much smaller than the number of samples (N).
While an outlier point results in N large distances (extremely small p ij values) in neighborhood embedding, there will be much fewer (K << N) of those large distances in cluster embedding.
Therefore, the effect of outliers on cluster embedding can be assumed to be much lower compared to the assumption in neighborhood embedding.
P ROPOSED M ETHOD
We propose a novel cluster embedding method, Gaussian Cluster Embedding in Autoencoder Latent Space (G-CEALS), by replacing the t-distribution (Equation 3) with a multivariate Gaussian distribution and the target distribution (Equation 4) with the Gaussian likelihood of individual tabular data samples ( X i ) belonging to a given cluster ( C j ) as P ( X i | C j ) or p ij .
Two clustering algorithms are used separately in training and evaluating the proposed joint cluster embedding method: 1) k-means and 2) Gaussian mixture model (GMM).
The clustering on tabular data space ( X i ∈ ℜ d ) yields K cluster assignments for individual samples.
Each cluster j is characterized by a centroid vector ( µ j ∈ ℜ d ) and a covariance matrix ( Σ j ∈ ℜ dxd ).
Because the dimensionality of tabular data is not as large as image data, Σ and µ parameters can be reasonably sized for computation.
Therefore, the soft cluster assignment ( S x ( i, j ) ) for individual samples can be obtained using a Gaussian kernel, which is the negative exponent of the Mahalanobis distance ( d x ( i, j ) ) between the point ( X i ) and the j -th cluster centroid vector, as shown in Equations 5 and 6.
[Equation]
To ensure that the sum of all soft cluster assignments equals one for a given sample, we obtain a joint cluster distribution, P ( x i , µ j ) or p ij , as shown in Equation 7.
We set p ij , obtained via superior traditional machine learning, as the target distribution to improve the autoencoder embedding ( z i ) of tabular data.
Similarly, the embedding distribution ( q ij ) can be obtained using soft Gaussian cluster assignments ( S z ) on the low-dimensional latent space ( z i ), as shown in Equation 7.
[Equation]
Therefore, our P and Q distributions are independent, unlike the current cluster embedding methods.
Additionally, the covariance of the target ( Σ x ) and embedding ( Σ z ) distributions can regulate the scatter or compactness of data clusters, which is impossible with t-distributed embedding.
A single-layer autoencoder is trained to encode the input (X) to a latent space (Z), which is then decoded to reconstruct the original input ( ˆ X i ), as shown in Equation 8. [Equation]
Input: d -dimensional tabular data, X ∈ ℜ n × d , where X i ∈ ℜ d
Output: Tabular data embedding, Z ∈ ℜ n × m , m <<
d j − th cluster parameters, { µ xj , Σ xj } ← K-means or GMM clustering of X p ij ← { µ xj , Σ xj } , in Equation Initialize: W 0 = { W encoder , W decoder } for t = 1 → n epochs do { ˆ X , Z t } ← Encoder ( X , W tencoder ) { µ zj , q ij } ← K-means or GMM clustering of Z t and using q ij in Equation L←
L auto + γ ∗ L cluster , measure the loss terms in Equation
W t ← AutoEncoder ( W t − 1 ), update weights minimizing the joint loss in Equationend for
Data set
[Table]
Here, θ and Φ denote the trainable parameters of the encoder and decoder, respectively.
The embedding obtained following each epoch of training is clustered using a clustering algorithm to obtain the cluster parameters ( µ , Σ ) and the embedding distribution ( q ij ).
Given the target distribution ( p ij ) (Equation 7), one of the learning objectives of G-CEALS is to minimize the KL divergence between
P and Q distributions ( L cluster ), as shown in Equation 9 below.
The overall learning objective of G-CEALS is to update the autoencoder’s weights by minimizing a joint cost function, the encoder reconstruction loss and the KL divergence loss, as below.
[Equation]
Here, γ is a trade-off hyperparameter to balance the contribution of cluster divergence L cluster during representation learning.
The G-CEALS algorithm is summarized in Algorithm 1.
E XPERIMENTS
All experimental steps and algorithms are implemented and evaluated in Python.
The neural networks are built using the PyTorch package, and clustering modules are developed using the sci-kitlearn package
We evaluate the proposed and baseline methods on seven multi-domain and multivariate tabular data sets.
A summary of these tabular data sets is provided in Table 1.
5. B ASELINE METHODS
All existing cluster embedding methods are benchmarked on image data sets, mainly using 2D filters in deep CNN architectures. Compared to 2D image data, limited sample size and dimensionality of tabular feature vectors will require a simpler architecture due to overfitting and 1D filters instead of 2D image filters. For example, the latent space size is set to 256 for image learning, whereas the input dimension of tabular data can be as low as 10.
We detail in Appendix B why existing methods for image learning are not suitable to apply directly on tabular data sets.
Considering
[Figure] [Figure] these factors, we compare our method against four baseline methods. First, the k-means and GMM clustering are performed on input tabular data (X) because traditional machine learning methods are known to produce competitive results on tabular data, unlike image data.
Second, a two-stage method is used: 1) the embedding (Z) extraction by training a single-layer autoencoder and then 2) perform clustering on Z (Peng et al., 2016; Tian et al., 2014).
Third, embedding learning and clustering are performed jointly on tabular data.
We compare fully-connected autoencoder (FCAE) and CNN autoencoder (similar to the DEPICT method Dizaji et al.
in single-layer and three-layer settings to investigate the best learning architecture for tabular data.
Fourth, despite the challenges in adapting image learning methods to tabular data learning (Appendix B), we use two pioneering methods for cluster embedding, DEC Xie et al.
(2016), IDEC Guo et al.
(2017), and a more recent method (deep k-means) DKM Moradi Fard et al.
(2020) for tabular data to compare with our proposed method.
Although deep and sophisticated autoencoder architectures proposed in the literature (stacked, variational, adversarial, convolutional) can be compared in an exhaustive search for the best method, it will introduce architectural bias in our claim for the best algorithm. Therefore, we use a common single-layer autoencoder to make a fair comparison between our and the baseline algorithms, especially considering the size and dimensionality of tabular data.
5. E VALUATION
The proposed G-CEALS model training involves self-supervised data reconstruction and unsupervised clustering without requiring any ground truth.
However, existing studies show that the hyperparameter ( γ in Equation 9) value is data-dependent and is tuned based on clustering accuracy that requires ground truth labels. The quality of cluster embedding is evaluated in downstream clustering tasks using two standard metrics: clustering accuracy (ACC) Kuhn (1955) and normalized mutual information (NMI) Est ´ evez et al.
We follow the same metrics for hyperparameter tuning and cluster embedding evaluation.
However, existing methods report the clustering accuracy on the same training data set because of the unsupervised nature of the problem. In contrast, we use a semi-supervised five-fold cross-validation scheme to obtain reproducible and transferable learning for downstream clustering or classification.
As shown in Figure 1, four data folds are used in unsupervised training and supervised tuning, which is then used to obtain the embedding of a left-out test data fold.
We report the average ACC and NMI scores across the five left-out data folds to compare the proposed and baseline methods.
These metrics score between 0 (failure) and 1 (perfect clusters).
For all evaluation purposes, the cluster number is set to the number of class labels for a given data set.
The scores are multiplied by 100 to represent the numbers in percentage.
R ESULTS
All experiments are conducted on a Dell Precision 5820 workstation running Ubuntu 20.04 with 64GB RAM and an NVIDIA GeForce RTX 3080 GPU with 10GB memory.
We standardize all tabular data using the mean and standard deviation of individual variables before training the autoencoder or performing clustering. [Table]
The DKM method is used on tabular data set without customizing this image learning method for tabular data.
Otherwise, a single-layer autoencoder without pre-training is used for all representation learning methods to avoid architectural bias in comparing the algorithms.
6. L EARNING ARCHITECTURE AND MODEL SELECTION
We compare the performance of fully connected autoencoders (FC-AE) and convolutional autoencoders (CNN-AE) in single- and three-layer architectures for our tabular data sets.
Table 5 in Appendix C clearly shows the superiority of single-layer FC-AE architecture over CNN and deeper architectures, which we select in our subsequent analysis.
A single-layer autoencoder maps a ddimensional tabular data sample to a five-dimensional autoencoder latent space (d > 5), considering the range of dimensionality of our tabular data sets (seven to 241).
For all experiments, the learning rate is set to 0.0001 with an Adam optimizer.
The best autoencoder model jointly trained clustering is selected by searching the best epoch point and γ value while training it for a maximum ofepochs.
The best gamma value is searched from a range between 0.1 and 1.0.
Table 6 in Appendix D shows example training epochs and gamma values that yield the best clustering accuracy on training data folds.
6. C LUSTERING OF TABULAR DATA VERSUS LATENT SPACE
Tables 2 and 3 show clustering scores and rank ordering for nine methods, respectively. Traditional clustering (K-means and GMM) on tabular data yields the top three scores for the dermatology, breast cancer, mice protein, and olive data sets.
This finding is at odds with the previous finding that direct clustering of images in pixel space yields the worst performance. This is because tabular data sets have relatively lower dimensionality and the absence of regularity in patterns makes such data still suitable for traditional machine learning.
For example, these clustering methods yield the worst ( < 1.0, max. 100) NMI scores for the highest dimensional (241) TUANDROMD data set. Alternatively, clustering methods (GMM, K-means) can be applied to the autoencoder’s latent space (Z).
A trained autoencoder is used to obtain the embedding on test data folds.
The test data embedding is then clustered using GMM and K-means, which are presented as GMM on Z and Kmeans on Z in Table 2, respectively.
Except for the E. coli data set, GMM on Z performs worse than GMM clustering of other data sets.
Similarly, K-means clustering of tabular data yields substantially better accuracy than K-means clustering of Z, except for the vehicle data set.
6. C LUSTERING OF JOINT CLUSTER EMBEDDING
The autoencoder latent space Z is jointly learned with data cluster distributions in this method.
Our results on tabular data are reproduced using two pioneering cluster embedding methods: DEC Xie et al.
(2016) and IDEC Guo et al.
The DEC method appears to be among the worst of nine methods presented in Table 2, except for the vehicle data set.
Therefore, a method proposed for image learning may not perform equally well on tabular data.
However, the improved DEC (IDEC) method shows substantial improvement over the DEC method.
The IDEC method always [Table] outperforms the GMM on Z except for the mice protein data set.
K-means clustering on Z is always inferior to the IDEC method.
The IDEC is the best of all methods for the olive data sets. Unlike DEC and IDEC methods, we do not modify the DKM method for the tabular data sets. Using the original image learning architecture (500-500-2000 neurons) on tabular data, the DKM method yields the worst of all clustering accuracy with a zero NMI score for the olive data set. However, it performs the best for the Ecoli data set with the lowest dimensionality (8) of all data sets.
6. P ROPOSED G-CEALS METHOD
Table 2 shows that our proposed G-CEALS method jointly trained with GMM clustering (ACC: 91.2) outperforms the second-best method, K-means clustering (ACC: 90.2), for the breast cancer data set.
The proposed G-CEALS method with GMM clustering (ACC: 77.2) also outperforms the second-best method, GMM clustering (ACC: 76.8) for the dermatology data set.
For the E. coli data set, the proposed method (ACC: 32.2) is on par with the second-best method, IDEC (ACC: 32.6).
The G-CEALS method (ACC: 83.4) outperforms the second-best IDEC (ACC:82.0) method on the
TUANDROMD data set.
The proposed method with GMM clustering (ACC: 42.0) again outperforms the second-best method, GMM clustering (ACC: 40.8), on the mice protein data set.
Only for the Olive data set, the IDEC method (ACC: 77.2) outperforms the proposed G-CEALS method with K-means clustering (ACC: 73.6).
However, the G-CEALS method with K-means clustering (ACC: 44.2) outperforms the second best method, IDEC (ACC: 42.8) on the vehicle data set.
Table 3 shows that the proposed G-CEALS method ranks the best method on four of the seven data sets.
For the other three data sets, G-CEALS ranks among the top three.
The average ranking reveals that our method (average rank: 1.4) substantially outperforms the second-best (IDEC, average rank: 3.0) and the third-best method: GMM clustering of tabular data (average rank: 3.9).
D ISCUSSION OF RESULTS
The paper is one of the first studies to jointly learn embedding and clustering in an autoencoder latent space with tabular data.
The findings of this article can be summarized as follows.
First, traditional clustering on tabular data is competitive with clustering on the autoencoder latent space.
Our method outperforms these superior methods for clustering.
Second, joint embedding learning with a cluster distribution (IDEC and our method) shows improved data representation over all disjoint learning methods (DEC, K-means on Z, GMM on Z) and GMM/K-means clustering on tabular data.
Including the DKM method, cluster embedding methods yield the best performance on all seven tabular data sets.
Third, our assumption of Gaussian clusters and an independent target distribution have improved the clustering accuracy over the methods using t-distributed clusters and all other baselines on average.
We elaborate on these findings in the following sections.
7. I MAGE VERSUS TABULAR DATA EMBEDDING
In computer vision, clustering images on high-dimensional image pixel space is ineffective, as reported in previous studies.
On the other hand, deep learning methods simultaneously and efficiently achieve dimensionality reduction, feature learning via convolution operations, and classification in an end-to-end framework.
In contrast, tabular data sets appear with relatively smaller sample sizes and dimensionality than image data and do not contain regularity in features to leverage the benefit of convolutional feature extraction.
Our results confirm that deep models with three hidden layers or convolutional networks or clustering on autoencoder embedding are not effective for tabular data.
This finding confirms the superiority of traditional machine learning on tabular data over deep learning. However, our proposed representation learning method outperforms the superior clustering method to demonstrate that tabular data need special algorithms to perform effectively with neural network-based learning architectures.
G-CEALS performs good with the GMM clustering algorithm may be due to the multivariate Gaussian modeling of the embedding distributions.
Clustering on autoencoder embedding (Z) is unsatisfactory because the reconstruction loss may have altered the cluster distribution in the latent space. Therefore, setting the superior cluster distribution on tabular data space as the target may have helped in improving the quality of the cluster embedding.
The G-CEALS method is designed for tabular data and may not be appropriate for images because the target distribution would be ineffective on pixel space.
Overall, the cluster embedding of tabular data requires a learning algorithm and architecture distinct from those proposed for image data.
7. T RADE - OFF BETWEEN DATA RECONSTRUCTION AND CLUSTER DIVERGENCE LOSS
We observe that one of the first successful deep cluster embedding methods proposed for image clustering (DEC Xie et al.
(2016)) performs the worst among all approaches with tabular data sets 3.
The DEC method first pretrains an autoencoder and then separately finetunes the KL divergence loss after modeling a t-distributed embedding for clustering.
There is no γ parameter because the L auto and L cluster optimizations happen separately in DEC.
Conversely, the introduction of γ parameter in IDEC and our method has substantially improved the quality of cluster embedding.
While the autoencoder reconstruction loss L auto retains all information in the latent space, L cluster (Equation 9) disrupts the latent space to learn the clusters.
A disproportionate contribution of these two loss terms can collapse the cluster distribution in the embedding leading to poor cluster performance.
Therefore, tuning the γ parameter is a crucial step in cluster embedding methods.
7. G-CEALS VERSUS OTHER METHODS
Our results show that the IDEC method performs better in cases where the clustering on tabular data space is substantially worse (E. coli and Olive data sets in Table 3).
It may be because the IDEC method does not learn from the cluster distribution on the tabular data space, similarly to the proposed G-CEALS method.
In IDEC, the target distribution is derived from the t-distributed embedding instead.
Overall, finding an appropriate target distribution for optimizing the KL divergence loss remains an open problem for future work.
Another common observation is that the Olive and E. coli data sets have the lowest dimensionality among the seven tabular data sets (Table 1) with only ten and seven variables, respectively.
Therefore, it may be inferred that the IDEC method is superior to the proposed G-CEALS method when the dimensionality of tabular data is less than ten.
For tabular data sets with the highest dimensionality ( TUANDROMD (241) and mice protein (78)), the
G-CEALS method outperforms the IDEC method as the best-performing method.
This observation suggests that Gaussian embedding may be more suitable over its t-distributed counterparts when a tabular data set has higher dimensionality.
C ONCLUSIONS
This paper presents a novel cluster embedding method in one of the first studies on tabular data sets.
The superiority of our G-CEALS method suggests that multivariate Gaussian distribution is superior to the widely used t-distribution assumption for learning tabular data embedding.
Our findings show that tabular data sets require learning algorithms and architectures distinct from those proposed for image learning.
This data-centric learning approach may improve a deep model’s performance on tabular data over its currently known superior machine learning counterparts. The proposed joint learning framework provides a promising representation learning of tabular data over its superior machine learning counterparts. A CKNOWLEDGMENTS
Acknowledgments are kept hidden for anonymity in double-blind review. A C LUSTER EMBEDDING VERSUS NEIGHBORHOOD EMBEDDING [Figure] [Figure]
P r o j e c t e d
Sp a c e
CNVDMEDrusenNormal
[Figure]
[Figure]
[Figure]
P r o j e c t e d
Sp a c e
CNVDMEDrusenNormal
[Figure]
t
[Figure]
DEC Xie et al.
(2016)/ Van Der Maaten & Hinton (2008)
IDEC Guo et al.
[Figure]
Neighborhood embedding
Cluster embedding
Low-dimensional
Sampled from Gaussian
Autoencoder embedding ( z i ) with low σlatent space
Distance or similarity
Between sample
Between point & cluster measure points ( x i , x j ) centroid ( x i , µ j )
Embedding t-distribution, t-distribution, distribution ( q ij ) α
=α =
Target
Gaussian in high-dimensional
A function of distribution ( p ij ) space (x) t-distributed q ij
Learning z
i +1 =
z i + d KLD ( p,q ) d ( z i ) w i
{++
+}
=
w i + d KLD ( p,q ) d ( w )
[Figure]
Visualization in d
[Figure]
Clustering in d >
B I MAGE VERSUS TABULAR DATA CLUSTER EMBEDDING METHODS
Caron et al.
learned visual features from images using AlexNet and VGG-16 after Sobel filtering for color removal and contrast enhancement, which do not apply to tabular data (Caron et al., 2018).
Their deepCluster architecture had five convolutional layers with up to 384 2D image filters to learn image texture.
Transfer learning of tabular data, similarly to VGG-16 on images, is not intuitive because tabular datasets (e.g., from health records data to criminal justice data) do not share common transferable textures or patterns like images.
Furthermore, these deep architectures can easily overfit data with limited sample size and dimensionality, similarly to tabular data sets.
The DEPICT method used a convolutional denoising autoencoder for reconstructing original images from corrupted images (Dizaji et al., 2017).
Because similar image corruption is not trivial on data tables, we used standard CNN autoencoders with single and three layers as baseline methods (Table 5 - Appendix C).
The deep clustering network (DCN) method used a fully-connected deep neural network (FC-DNN) with 2000, 1000, 1000, 1000, and 50 neurons for learning high-dimensional image data (Yang et al., 2017), whereas tabular data can have as low as ten features.
They avoided
CNN architecture for image learning - ”to concentrate on their DCN algorithm to provide a proofof-concept rather than exhausting the possibilities of combinations”.
However, they leveraged the architectural benefit of a stacked deep autoencoder instead of using a regular autoencoder model, which can mask the original contribution of the algorithm.
Similarly, the deep k-means (DKM) method used FC-DNN instead of CNN for image learning (Moradi Fard et al., 2020).
The authors of DKM compared their method only against the methods that used FC-DNN (excluding all CNN based methods) to avoid architectural bias.
However, we use the original DKM method to reproduce cluster embedding on tabular data.
The suboptimal performance of DKM in Tables 2 and 3 suggests the need for customizing these architectures for learning tabular data.
Recently, Mrabah et al.
, in their Dynamic Autoencoder (DynAE) method, used image augmentation (shifting and rotation), which is not intuitive with tabular data, and a 2D convolutional adversarial autoencoder for image data, which needs substantial customization of the adversarial network for learning 1D feature vectors in tabular format (Mrabah et al., 2020).
One main limitation of the DynAE and DKM methods is that the latent dimension is restricted to the number of clusters, whereas our method is proposed for any latent dimension.
In general, the architectural benefits include a) stacked vs. variational vs. adversarial vs. convolutional autoencoders, b) CNN vs. DNN, c) with vs. without pretraining, d) deep vs. shallow are explored to exhaustively search the best model, which do not provide a fair ground to prove an algorithm’s performance or the hypothesis regarding the proposed embedding distribution.
Therefore, to focus on our hypothesis and algorithm contributions at this early stage of tabular data learning research, we avoid exhaustive model search using special autoencoders (stacked, variational, adversarial, convolutional), model pretraining, data augmentations, and deepening of the network.
Therefore, we used the same single-layer standard autoencoder architecture to fairly compare against the baseline algorithms.
C C OMPARISON BETWEEN LEARNING ARCHITECTURES .
[Table]
D T UNING MODEL HYPERPARAMETERS
[Table]