SuperTML: Domain Transfer from Computer Vision to Structured Tabular Data through Two-Dimensional Word Embedding
Abstract
Structured tabular data is the most commonly used form of data in industry according to a
Kaggle ML and DS Survey.
Gradient Boosting Trees, Support Vector Machine, Random Forest, and Logistic Regression are typically used for classiﬁcation tasks on tabular data.
The recent work of Super Characters method using twodimensional word embeddings achieved state-ofthe-art results in text classiﬁcation tasks, showcasing the promise of this new approach.
In this paper, we propose the SuperTML method, which borrows the idea of Super Characters method and two-dimensional embeddings to address the problem of classiﬁcation on tabular data.
For each input of tabular data, the features are ﬁrst projected into two-dimensional embeddings like an image, and then this image is fed into ﬁnetuned ImageNet CNN models for classiﬁcation.
Experimental results have shown that the proposed SuperTML method have achieved state-ofthe-art results on both large and small datasets.
1. Introduction
In data science, data is categorized into structured data and unstructured data.
Structured data is also known as tabular data, and the terms will be used interchangeably.
Anthony
Goldbloom, the founder and CEO of Kaggle observed that winning techniques have been divided by whether the data was structured or unstructured (Vorhies, 2016).
Currently, DNN models are widely applied for usage on unstructured data such as image, speech, and text.
According to Anthony, “When the data is unstructured, its deﬁnitely CNNs and RNNs that are carrying the day” (Vorhies, 2016).
The successful CNN model in the ImageNet compe- > tition (Russakovsky et al., 2015) has outperformed human for image classiﬁcation task by ResNet (He et al., 2016) since 2015.
On the other side of the spectrum, machine learning models such as Support Vector Machine (SVM), Gradient Boosting Trees (GBT), Random Forest, and Logistic Regression, have been used to process structured data.
According to a recent survey of 14,000 data scientists by Kaggle (2017), a subdivision of structured data known as relational data is reported as the most popular type of data in industry, with at least 65% working daily with relational data.
Regarding structured data competitions, Anthony says that currently XGBoost is winning practically every competition in the structured data category (Fogg, 2016).
XGBoost (Chen & Guestrin, 2016) is one popular package implementing the Gradient Boosting method.
Recent research has tried using one-dimensional embedding and implementing RNNs or one-dimensional CNNs to address the TML (Tabular Machine Learning) tasks, or tasks that deal with structured data processing (Lam et al., 2018; Thomas, 2018), and also categorical embedding for tabular data with categorical features (Guo & Berkhahn, 2016; Chen et al., 2016).
However, this reliance upon onedimensional embeddings may soon come to change.
Recent NLP research has shown that the two-dimensional embedding of the Super Characters method (Sun et al., 2018) is capable of achieving state-of-the-art results on large dataset benchmarks.
The Super Characters method is a two-step method that was initially designed for text classiﬁcation problems.
In the ﬁrst step, the characters of the input text are drawn onto a blank image.
In the second step, the image is fed into two-dimensional CNN models for classiﬁcation.
The two-dimensional CNN models are trained by ﬁne-tuning from pretrained models on large image dataset , e. g.
ImageNet. In this paper, we propose the SuperTML method, which borrows the concept of the Super Characters method to address TML problems.
For each input, tabular features are ﬁrst projected onto a two-dimensional embedding and fed into ﬁne-tuned two-dimensional CNN models for classiﬁcation.
The proposed SuperTML method handles the categorical type and missing values in tabular data automati- cally, without need for explicit conversion into numerical type values.
2. The Proposed SuperTML Method
The SuperTML method is motivated by the analogy between TML problems and text classiﬁcation tasks.
For any sample given in tabular form, if its features are treated like stringiﬁed tokens of data, then each sample can be represented as a concatenation of tokenized features.
By applying this paradigm of a tabular sample, the existing CNN models used in Super Characters method could be extended to be applicable to TML problems.
As mentioned in the introduction, the combination of twodimensional embedding (a core competency of the Super
Characters methodology) and pre-trained CNN models has achieved state-of-the-art results on text classiﬁcation tasks.
However, unlike the text classiﬁcation problems studied in (Sun et al., 2018), tabular data has features in separate dimensions.
Hence, generated images of tabular data should reserve some gap between features in different dimensions in order to guarantee that features will not overlap in the generated image.
SuperTML is composed of two steps, the ﬁrst of which is two-dimensional embedding.
This step projects features in the tabular data onto the generated images, which will be called the SuperTML images in this paper.
The conversion of tabular training data to SuperTML image is illustrated in Figure 1, where a collection of samples containing four tabular features is being sorted.
The second step is using pretrained CNN models to ﬁnetune on the generated SuperTML images.
Figure 1 only shows the generation of SuperTML images for the training data.
It should be noted that for inference, each instance of testing data goes through the same preprocessing to generate a SuperTML image (all of which use the same conﬁguration of two-dimensional embedding) before getting fed into the CNN classiﬁcation model.
Considering that features may have different importance for the classiﬁcation task, it would be prudent to allocate larger spaces for important features and increase the font size of the corresponding feature values.
This method, known as SuperTML VF, is described in Algorithm 1.
To make the SuperTML more autonomous and remove the dependency on feature importance calculation done in Algorithm 1, the SuperTML EF method is introduced in Algorithm 2.
It allocates the same size to every feature, and thus tabular data can be directly embedded into SuperTML images without the need for calculating feature importance.
This algorithm shows even better results than 1, which will be described more in depth later in the experimental section.
3. Experiments
The data statistics from UCI Machine Learning Repository is shown in Table 1.
3.1. Experiments on the Iris dataset
“This is perhaps the best known database to be found in the pattern recognition literature” 1 .
The Iris dataset is widely used in machine learning courses and tutorials.
Figure 2a shows an example of a generated SuperTML image, created using Iris data.
The experimental results of using SEnet-154 shown in Table 2 is based on an 80:20 split of the 150 samples.
It shows that the proposed SuperTML method achieves the same accuracy as XGBoost on this small dataset.
ml/datasets/Iris
Input : Tabular data training set
Parameter :
Image size of the generated SuperTML images
Output : Finetuned CNN model 1: for each sample in the tabular data do 2: for each feature of the sample do 3:
Draw the feature in the same font size without overlapping, such that the total features of the sample will occupy the image size as much as possible.
4: end for 5: end for 6: Finetune the pretrained CNN model on ImageNet with the generated SuperTML images.
7: return the trained CNN model on the tabular data
[Figure]
3.2. Experiments on the Wine dataset
For this dataset 2 , we use SuperTML VF, which gives features different sizes on the SupterTML image according to their importance score.
The feature importance score is obtained using the XGBoost package (Chen & Guestrin, 2016).
One example of a SuperTML image created using data from this dataset is shown in Figure 2b.
The results in Table 2 shows that the SuperTML method obtained a slightly better accuracy than XGBoost on this dataset.
3.3. Experiments on the Adult dataset
The task of this Adult dataset 3 is to predict whether a persons income is larger or smaller than 50,000 dollars per year based on a collection of surveyed data.
For categorical features that are represented by strings, the
Squared English Word (SEW) method (Sun et al., 2019)
[Figure]
[Table]
[Table]
is used.
One example of a generated SuperTML image is given in Figure 3.
Table 2 shows the results on Adult dataset.
We can see that on this dataset, the SuperTML method still has a higher accuracy than the ﬁne-tuned XGBoost model, outperforming it by 0.32% points of accuracy [Table]
3.4. Experiments on the Higgs Boson Machine Learning Challenge dataset
The Higgs Boson Machine Learning Challenge involved a binary classiﬁcation task to classify quantum events as signal or background.
It was hosted by Kaggle, and though the contest is over, the challenge data is available on opendata (Adam-Bourdarios et al., 2015).
It has 25,000 training samples, and 55,000 testing samples.
Each example hasfeatures, each of which is stored as a real number value.
In this challenge, AMS score (Adam-Bourdarios et al., 2014)
[Figure]
[Table]
is used as the performance metric.
Figure 4 shows two examples of generated SuperTML images.
Table 3 shows the comparison of different algorithms.
The
DNN method and XGBoost used in the ﬁrst two rows are using the numerical values of the features as input to the models, which is different from the SuperTML method of using two-dimensional embeddings.
It shows that SuperTML EF method gives the best AMS score of 3.979.
In addition, the SuperTML EF gives better results than SuperTME VF results, which indicates SuperTML method can work well without the calculation of the importance scores.
4. Conclusion
The proposed SuperTML method borrows the idea of twodimensional embedding from Super Characters and transfers the knowledge learned from computer vision to the structured tabular data.
Experimental results shows that the proposed SuperTML method has achieved state-of-the-art results on both large and small tabular dataset.