T AB C APS : A C APSULE N EURAL N ETWORK FOR T ABULAR D ATA C LASSIFICATION WITH B O W R OUTING
Anonymous authors
Paper under double-blind review
A BSTRACT
The instances in a table are represented by a collection of heterogeneous tabular features.
Previous work often made predictions for such instances in a paradigm that processed tabular features as operating units, which requires to well cope with the heterogeneity.
In this paper, we propose to encapsulate all tabular features of an instance into vectorial features and process them collectively rather than have to deal with individual ones, which directly captures the representations at the instance level and benefits robust performances.
Specifically, we adopt capsules to organize tabular features of the instance into vectorial features, and devise a novel capsule neural network called T AB C APS to process the vectorial features for classification.
In T AB C APS , a tabular instance is respectively encoded into several vectorial features by some optimizable multivariate Gaussian kernels in the primary capsule layer, where each vectorial feature represents a specific profile of the input instance and is transformed into senior capsule layer under the guidance of a novel straightforward routing algorithm.
The design of routing algorithm is motivated by the Bag-of-Words (BoW) model, which performs capsule feature grouping straightforwardly and efficiently, in lieu of the computationally complex clustering of previous routing algorithms.
Comprehensive experiments show that T AB C APS achieves competitive and robust performances in tabular data classification tasks.
I NTRODUCTION
Tabular data are ubiquitous in real world applications, which records abundantly meaningful information such as medical examination results (Hassan et al., 2020) and company financial statements
(Addo et al., 2018).
Previous methods often processed a tabular instance by treating its tabular features as the operating units.
For example, decision tree based methods (Breiman et al., 1984; Chen & Guestrin, 2016) used one tabular feature in each decision step, and neural networks (Gorishniy et al., 2021;
Chen et al., 2022) elaborately executed feature-wise interactions to capture higher-level semantics.
However, tabular features might not be capable to represent comprehensive semantics due to the scanty representation spaces of their scalar formats, and it is intractable to design tailored featurewise interaction approaches in deep learning frameworks (Arik & Pfister, 2020; Chen et al., 2022;
Gorishniy et al., 2021) due to the heterogeneity of these features.
In this paper, we propose a novel paradigm for tabular learning, which encapsulates all tabular features of an instances into vectorial features and directly treats the vectorial features as operating units.
Such design utilizes the sufficient representation space of the vectorial feature format to probably learn the comprehensive instance-level semantics, which avoids executing complex interactions among heterogeneous features.
To this end, we borrow the concept of capsules (Sabour et al., 2017) to organize vectorial features, and propose a novel capsule neural network (CapsNet) called T AB C APS for tabular data classification.
In T AB C APS , several optimizable multivariate Gaussian kernels encode tabular instances into the primary capsules, whose features (in vector format) represent the marginal likelihoods of the input instance w.r.t.
the corresponding Gaussian distributions.
We set the scale and location parameters of these multivariate Gaussian kernels learnable, thus allowing these kernels to model some plausible “instance types” and each primary capsule feature represents a specific profile of the input instance that measures the likelihood to these “instance types”.
Unlike previous CapsNets that used one senior capsule to predict the probability of the input instances belonging to one class, we use multiple senior capsules for a class, motivated by ensemble learning.
In previous CapsNets, primary capsule features
Tabular
[Figure]
Concatenation
[Figure]
[Figure]
P.C. Features
S.C. Features … … … … …
Sparse Projection by Entmax 1.5 (𝑊 𝑗 )
P.C. Features
Initialized S.C. Features 𝑣′ 𝒋
P.C. Votes ො𝑢 𝒋|∙ …
Computing Average
P.C. Votes Selecting
Features Updating of Final S.C.
[Figure]
BoW
[Figure]
Alg
[Figure]
𝑠 2′
[Figure]
1′ 𝑠
[Figure]
Words’
[Figure]
were transformed into senior capsules by an affinity projection and a routing algorithm (Sabour et al., 2017; Hinton et al., 2018) that groups similar primary capsule features by clustering processes.
In T AB C APS , a novel sparse projection method and a novel straightforward routing algorithm are proposed to perform the feature transformation from primary to senior capsules.
Our proposed routing algorithm is much more efficient than previous routing algorithms, since the primary capsules in previous CapsNets captured some unknown semantics from data (e.g., images) and had to apply feature clustering (an iterative process) to attain higher-level semantics.
In our
T AB C APS , each of our primary capsules represents the likelihood of the input instance w.r.t.
the
Gaussian distributions, which represents stable semantics and allows to perform a straightforward routing algorithm.
Motivated by the bag-of-words (BoW) model (Salton & Lesk, 1965) which is efficient in similar information search (Pineda et al., 2011), we propose a straightforward routing (called BoW Routing ) for T AB C APS .
The proposed BoW Routing computes similarities between primary capsule features and the initialized senior capsule features by counting the co-occurrences of “words” (implemented to be learnable templates), and incorporates those primary capsule features whose similarities are beyond an adaptive threshold to update the initialized senior capsule features.
Similar to previous CapsNets, we also present a special decoder, which is tailored for tabular feature reconstruction tasks (e.g., missing value imputation).
Contributions .
(i)
For the first time, we propose to encapsulate all the independent tabular features of a tabular instance into vectorial features as operating units, which avoids interactions between heterogeneous tabular features and directly learns the instance-level semantics.
(ii)
We propose a
CapsNet tailored for tabular learning, in which we devise a new primary capsule type with learnable multivariate Gaussian kernels and conduct a novel straightforward routing to synthesize senior capsule features at low cost.
(iii) Experiments on real-world datasets validate that T AB C APS attain robust performances in tabular data classification.  R ELATED W ORK
Capsule Neural Networks.
Capsule neural networks were first proposed to deal with image data, where a capsule uses a vector or matrix to represent an object (part), given the input instance.
The primary capsules capture the basic object parts by using templates (Kosiorek et al., 2019), optical flow analysis (Sabour et al., 2021), or location-based operations (Hinton et al., 2018). In the standard procedure of CapsNets, a primary capsule feature is treated as a unit and is linearly projected so as to transpose the pose of object parts. Then, a routing algorithm groups the projected capsule features (called feature votes ) to synthesize senior capsule features (e.g., representing object wholes). Most routing algorithms performed iteration processes at high computational costs (Sabour et al., 2017; Hinton et al., 2018) to conduct the “route-by-agreement”, since the primary capsules handle different semantics in dealing with different samples.
To avoid iterations, some straightforward routing algorithms were proposed (Choi et al., 2019; Ahmed &
Ribeiro et al., 2020).
However, these straightforward routing algorithms were for image tasks and some of them did not performed the part-to-whole learning.
At present, CapsNets have been widely used for image processing (Rajasegaran et al., 2019; Hinton et al., 2018; Chen et al., 2021), point clouds (Zhao et al., 2019b; 2020), recommendation in shopping (Li et al., 2019), and language tasks (Zhao et al., 2019a).
However, as far as we know, no bespoke CapsNets were proposed for tabular learning.
Notably, this paper presents a novel CapsNet for tabular data, in which a primary capsule learns a specific profile of the input tabular instance rather than a “part” of the instance.
Tabular Learning.
Various methods were proposed for tabular data processing, mainly including decision tree algorithms (Breiman et al., 1984; Quinlan, 1979; Chen & Guestrin, 2016;
Prokhorenkova et al., 2018; Ke et al., 2017) and neural network (NN) based methods (Arik & Pfister, 2020; Popov et al., 2019; Chen et al., 2022; Abutbul et al., 2021).
Different from tree based methods that split tabular feature spaces by some information metrics (Quinlan, 1979; 2014; Breiman et al., 1984) and sought for possible decision paths, NN based methods often introduced attention and grouping mechanisms (Chen et al., 2022; Arik & Pfister, 2020;
Guo et al., 2017; Wang et al., 2017; Gorishniy et al., 2021) for tabular feature interactions to yield high-level features.
However, tabular features in a table are heterogeneous and are relation-agnostic, and thus it is hard to conduct all the feature interactions in the right way.
M ETHODOLOGY
We propose to classify each tabular instance by processing a vectorial feature containing all its tabular feature values.
See the pipeline of our T AB C APS in Fig. 1(a), to better depict a tabular instance, its raw tabular features are first processed by a simple function (e.g., a fully connected layer) to yield more comprehensive tabular features, which are then concatenated with raw features and are jointly organized into the basal vectorial feature.
Then, this vectorial feature is mapped into primary capsule features (in a vectorial feature format) by various learnable multivariate Gaussian kernels.
Each primary capsule feature is transformed into feature votes for senior capsules by the proposed sparse projection (see Sec. 3.2.1), and senior capsule features are finally obtained by selecting and merging feature votes , under the guidance of the BoW Routing (see Sec. 3.2.2).
3. H OW TO E NCODE I NSTANCES INTO P RIMARY C APSULES ?
Unlike the known CapsNets for images that build primary capsules to represent different object parts respectively, in T AB C APS , different primary capsules represent different profiles for one instance.
Formally, an instance represented by a vectorial feature x ∈ R m is transformed into a feature u i ∈
R m of the i -th primary capsule by a multivariate Gaussian kernel k i , as:
[Equation]
where the location parameter vector µ i ∈ R m and the scale parameters Σ
i = diag ( σ 1 , σ 2 , . . .
, σ m ) of the multivariate
Gaussian kernel k i are set to be learnable, so that T AB C APS allows to make proper multivariate Gaussian kernels adaptive to the given tabular data.
Since scalar tabular features are typically regarded as independent, we set Σ as a diagonal matrix.
After Eq.
(1), a layer normalization is performed on each u
i ( i = 1 , 2 , . . . , N p , N p is the count of the primary capsules) for normalization following Tsai et al.
Specially, we also encapsulate the original tabular feature values (denoted by x ) to be an additional capsule feature.
This design can be interpreted that a primary capsule models an “instance type” by determining the parameters µ, Σ of the multivariate Gaussian kernels, and various primary capsule features of an input instance are represented as the likelihoods in reference to these “instance types”.
It is beneficial to find some “instance types”, which allows to implicitly model the tabular feature correlations.
3. H OW TO S YNTHESIZE S ENIOR C APSULE F EATURES ?
Unlike our primary capsules that model various feasible “instance types”, we set the senior capsules to learn the semantics of the target classes.
Motivated by ensemble learning, we use multiple senior capsules to learn the semantics for one target class, and obtain the final prediction by voting.
To attain the senior capsule features, primary capsule features are transformed by a sparse weight matrix into feature votes , and a newly proposed BoW routing selects feature votes to synthesize senior capsule features.
Note that the scalar features (vectorial feature components) are only linearly transformed, and the non-linear mapping (the routing) is performed by taking vectorial features as operating units.
3.2. S PARSE P ROJECTION
The affinity projection in previous work (Sabour et al., 2017) uses a learnable weight W j |
i ∈ R m × n to linearly map the i -th primary capsule feature u i ∈ R m to yield a feature vote ˆ u j | i ∈ R n for the j -th senior capsule, by ˆ u j |
i
=
u
i W j | i .
We modify the weight matrix by adding a sparse operation entmax (Peters et al., 2019), by: [Equation]
[Equation]
where α = 1 . 5 as default.
In this way, the weight matrix W ′ j | i is learned to be sparse, so that only some elements of u j | i is used to synthesize the feature votes and the target-irrelevant elements are excluded.
Different from sparse attention operations, the entmax is applied to the weight matrix.
Thus, the capsule feature component selection is consistent across different instances.
3.2. B O W R OUTING
Given the synthesized feature votes ˆ u j |
i for the j -th senior capsule (where i = 1 , 2 , . . .
, N p , j = 1 , 2 , . .
. , N s , N p and N s are the counts of primary capsules and senior capsules respectively), we use a routing algorithm to synthesize senior capsule features by selecting and merging ˆ u j | i .
Note that the known routing algorithms were implemented by clustering (Sabour et al., 2017; Hinton et al., 2018), since in previous work, a primary capsule may capture any object parts and thus the initialized senior capsule features (cluster centroids) are unknown before the routing step.
Different from that, each primary capsule in our design learns concrete semantics that represent the marginal likelihoods of a specific multivariate Gaussian distribution (i.e., a specific “instance type”), making it possible to predict the initialized senior capsule features directly using a fix function with respect to the primary capsule features.
We predict the initialized senior capsule features (the cluster centroids) v ′ j by directly using the average of the feature votes ˆ u j | i by v ′ j = (cid:80) N p
i ˆ u j | i /N p .
To select some feature votes for senior capsule feature synthesis, we measure the similarities between the initialized senior capsule feature v ′
j and the primary capsule feature vote
ˆ u j | i .
An intuitive solution is to compute the similarity directly.
However, computing similarity in the original feature space might hinder the performance, while a feature space related to the prediction target should be more beneficial.
Motivated by the success of the bag-of-words (BoW) model in similarity measure (e.g., information retrieval), we introduce the BoW model to the routing algorithm.
A classical BoW model first clusters the given features, and collects a word dictionary whose words are the centroids of feature clusters.
Then, a feature can be quantized as the “occurrence” of various words, and feature similarities can be computed by counting the co-occurring words.
In this way, the similarity is computed in the semantic space guided by the words.
While the traditional BoW model is performed in separated steps, we need a differentiable version for our routing algorithm.
Occurrence status computing.
The dictionary words in the traditional BoW model are computed according to the extracted features. Thus, in our routing we directly implement the word dictionary with a learnable parameter set of size M , { s m | m = 1 , 2 , . . .
, M } , with each “word” s m ∈ R n (of the identical size as the feature votes ˆ u j | i ).
To compute the “occurrences” of the “words”, an occurrence status (a scalar) between ˆ u j | i and the initialized senior capsule feature v ′ j for the “word” s m is defined by:
[Equation]
where p j | i ; m and p j ; m represent the degrees (i.e., the occurrence statuses) in which u T j | i and v ′
j T reveal the semantics of s m , we let p j | i
=
[ p j | i ;1 , p j | i ;2 , . .
. , p j | i ; M ] and p j =
[ p j ;1 , p j ;2 , . .
. , p j ; M ] represent semantic codes of ˆ u j | i and v ′ j .
In computing Eq.
(3), the “word” s m is normalized into s ′ m by the l 2 norm to avoid the impact of the vector lengths.
In traditional BoW models, an occurrence status is represented as binary (0 or 1), but in our routing, we generalize the discrete binary status to a continuous status form (in (0 , 1) ) using the “sigmoid” function.
Feature vote merging.
To compute the similarity between the occurrence statuses p j | i and p j , traditional BoW models use the Jaccard distance.
Since the occurrence statuses are continuous values in our routing, to which the Jaccard distance cannot be simply applied, we compute the pairwise distances
d j | i between p j | i and p j using the squared Euclidean distance function D :
[Equation]
Note that the squared Euclidean distance has been proved to be an effective approximation of the
Jaccard distance (Reina et al., 2014).
Based on the pairwise distances d j | i of feature votes and initialized senior capsule features, we can directly obtain the neighboring feature votes ˆ u j | i to update the features in the j -th senior capsule.
An obvious way is to perform k -Nearest Neighbor ( k -NN) search.
However, we find that k -NN can lead to instability in training, since the distributions of feature votes are unknown and k -NN might select some feature votes with extreme values.
To gain training stability, we propose to use an adaptive threshold to determine a range around the initialized senior capsule feature v ′
j (see Fig. 1(b)).
We set this threshold as a learnable positive value γ 2 > 0 , and the selection of feature votes is determined by:
[Equation]
where ε ( · ) is the Heaviside function, and if and only if q j | i = 1 , it indicates that the feature vote from the i -th primary capsule, ˆ u j | i , is selected.
Then, we compute the weights for the feature votes and obtain the final senior capsule feature v j by a weighted sum, as:
[Equation]
where the “softmax” operation guarantees all w j | i > 0 (giving small weights to unselected votes ), avoiding possible failure if no feature vote is selected to yield v j .
In this way, our proposed BoW routing performs a straightforward process to obtain the senior capsule features.
Some recent routing algorithms for images also performed straightforward processes based on some delicate designs using agglomerative hierarchical clustering (Chen et al., 2021) or variational bayes (Ribeiro et al., 2020).
Distinct from those methods, the efficiency of our BoW routing partially results from the innovative definition of the primary capsules that learns concrete semantics, allowing to predict initialized senior capsule features directly, in lieu of learning unknown object parts and requiring clustering-like routing algorithms to group them as in previous CapsNets.
3. S ENIOR C APSULE V OTING FOR C LASSIFICATION
Similar to the known CapsNets, we let the vector lengths of the senior capsule features represent the existence probabilities of one semantics, and this work models each target class by multiple senior capsules motivated by ensemble learning.
Suppose there are K classes.
We divide all the senior capsules into K groups, such that the features in each group vote for one particular class.
The classification outcomes are defined by:
[Equation]
where the vector c represents the predicted probabilities for all the K classes, and ˆ y denotes the predicted label.
A value l k in Eq.(7) denotes the sum of the vector lengths of the senior capsule features for the k -th class (i.e., in the group G k ), which is defined by: l k = (cid:80)
j ∈ G k
|| v j || 2 .
In training, we use the margin loss following Sabour et al.
(2017), as:
[Equation]
where T k = 1 if the k -th class is the target, and otherwise T k = 0 .
We set t + = 0 .
9 , t − = 0 .
1 , and λ = 0 .
5 following Sabour et al.
3. A D ECODER FOR T ABULAR F EATURE R ECONSTRUCTION
We build a decoder based on the Abstract Layers (Chen et al., 2022) for tabular feature reconstruction.
We stack two Abstract Layers in sequence with a shortcut performing feature concatenation, as shown in Fig.
Finally, a fully connected layer is on top of the decoder to produce the reconstructed tabular features.
Since there are multiple senior capsules voting for a target class, the decoder input is the concatenation of features in the group of senior capsules that votes for the correct class.
Commonly, we do not use the decoder as regularization like previous CapsNets (Sabour et al., 2017) and T AB C APS can perform well (see Fig. 3).
The decoder is used when we attempt to reconstruct raw features for some tasks (e.g., missing value imputation), which is trained under the specification of the mean square error (MSE), and MSE is scaled down by 5 × 10 − 6 and is jointly used with the loss L (in Eq.
E XPERIMENTS
In this section, we conduct comprehensive experiments to evaluate the effect of our T AB C APS by comparing with state-of-the-art methods.
We also present ablation studies to analyze the effects of some key components of T AB C APS , including our proposed decoder and the BoW routing .
4. E XPERIMENTAL S ETUPS
[Figure]
Diabetes (Ukani, 2020), EEG Eye State (EEG) (Dua & Graff, 2015), Gas Concentrations (Gas) (Vergara et al., 2012), and HillValley Dataset (Hill) (Lee & Franz, 2008).
These datasets have few or no categorical features, eliminating the impacts of various feature embedding approaches for categorical features in model comparison.
In pre-processing, the data are standardized by the z-score normalization.
For Click , we follow the train-test split and the data pre-processing in (Popov et al., 2019).
For the other datasets, we randomly divide 80% of the data into the training sets, and the rest 20% for the test sets.
We further sample 15% from the training sets for validation.
Experiments were run for 5 times with regard to random seeds.
Implementation details and comparison baselines.
Our method is implemented with PyTorch on Python 3.7, and is run on GeForce RTX 2080 Ti.
In training, we set the batch size as 2,048.
We use the QHAdam optimizer (Ma & Yarats, 2019) with weight decay rate 10 − 5 , v = (0 . 7 , 0 . 99) , and β = (0 .
We compare T AB C APS with several baseline models, including TabNet (Arik & Pfister, 2020), NODE (Popov et al., 2019), TF-Transformer (FT-T) (Gorishniy et al., 2021),
DANets (Chen et al., 2022), Net-DNF (Abutbul et al., 2021), XGboost (Chen & Guestrin, 2016),
CatBoost (Prokhorenkova et al., 2018), a fully connected neural networks (FCNN) (Nair & Hinton, 2010), the Vector CapsNet (Sabour et al., 2017), and FCNN with lasso regularization and mixup regularization ( α = 1 . 0 ) respectively.
The Vector CapsNet (Sabour et al., 2017) was designed for images; we replace its convolution layer by a fully connected layer for tabular data.
Following Popov et al.
; Arik & Pfister (2020), the performances of all the methods are hyperparameter-tuned on the validation sets (e.g., by grid search).
The hyperparameter search spaces and search algorithms are given in Appendix A.1.
We employ the “log-loss” metric to measure the model performances, which is considered as a more sensitive metric than accuracy, and is widely used in tabular data tasks (e.g., Kaggle competition).
All the performances of the other methods are obtained by running their original codes.
Our code will be open upon acceptance.
4. C LASSIFICATION P ERFORMANCE C OMPARISON
The classification performances are reported in Table 1.
One can see that our T AB C APS obtains the best or second best performances among the state-of-the-art methods, and is especially superior on some datasets by clear margins.
In addition, we also report the computational complexity and efficiency on Diabetes , using the measures of model size (# param.) and fps (the amount of instances [Table] [Table] that a model processes per second).
Since XGboost and CatBoost are traditional machine learning models, we do not report their complexity.
One can see that T AB C APS is much more efficient and greatly lighter (over 10 × ) than the other state-of-the-art methods (e.g., FT-Transformer, Net-DNF, and DANet-24), and significantly outperforms the models with similar model complexity (e.g., FCNN with regularization methods and Vector CapsNet).
Besides, the comparison with Vector CapsNet suggests the advance of our design for tabular data, with lower complexity and higher performances.
4. E XTREME G ENERALIZATION T EST
To further examine the generalization capability of T AB C APS , we attempt to test it on the training data and test data with very dissimilar feature values.
We conduct t-SNE projections (with default hyper-parameters of the “sklearn” python package 1 ) on all the instances of the 6 datasets, and make the train-test split based on the t-SNE embedding.
See the 2nd row in Table 2.
We use the instances inside the black rectangles for training (containing ∼ 70% data) and the rest for test.
Notably, the rectangles are biasedly placed, and the instances for training and test do not overlap in the t-SNE embedding and are clearly in different distributions.
The performances reported in Table 2 show that our T AB C APS obtains best or second best performances in deep learning approaches on all thedatasets.
Such results imply that our method can learn robust features and generalize well.
Besides, our T AB C APS exhibits better generalization than the Vector CapsNet with very clear margins (e.g.,
T AB C APS obtains 8 .
750 × 10 − 2 on EEG dataset, while the Vector CapsNet obtains 163 .
193 × 10 − 2 ).
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] CatBoost. 4. P ERFORMANCE S TABILITY D URING T RAINING Many previous work required the early stop (Yao et al., 2007) to avoid over-fitting because they managed to exploit the potential of each tabular feature, repeatedly.
However, our T AB C APS does not seem to require such training trick since it learns the instance-level semantics directly.
To show this, we compare the real-time validation performances (on log-loss values) during traning, among several regularization strategies (e.g., FCNN with lasso, mixup (Zhang et al., 2018)), common neural networks (e.g., TabNet with sparse regularization, NODE and Net-DNF with ensemble learning strategies), and traditional boosting methods (XGboost and CatBoost), on the Click dataset.
As shown in Fig.
3, most these methods obtain their lowest log-loss values on the validation set after few training steps, and then their models gradually collapse as the training step increases.
Notably, during the training process, all the corresponding loss values on training data keeps on decreasing or slightly fluctuating.
One can see that the lasso and mixup regularization can alleviate this issues to some extent, but the validation log-loss values still increase slowly.
Different from these methods, our
T AB C APS with and without the decoder performs evidently stable, stabilizing the log-loss values at a low level (around 0 . 62 ).
This finding reveals an advantage of using vectorial features as operating units and suggests that our method can facilitate the users to obtain a well-performed model without intensely monitoring the performances on the validation sets or agonizing over the use of training tricks (e.g., regularization methods or early stop).
4. E FFECTS OF B O W R OUTING
Here we further inspect how our proposed BoW Routing determines the transformation paths for the vectorial features from primary capsules to senior capsules.
We compute the averaged transformation weights ( w j | i in Eq.(6)) over the whole test set and visualize them as in Fig.
One can see that a senior capsule features are yielded by merging limited vectorial feature votes from primary capsules.
Besides, we found the weights are sparse and the values are similar to different samples (with the standard deviation values varying from 0 . 001 to 0 .
003 ), which suggests that primary capsules model some semantically meaningful “instance types” and our BoW routing succeeds in finding out clear decision paths for classification.
4. A BLATION S TUDIES
We conduct ablation studies to examine the effects of the routing algorithms and the decoder (in a missing value imputation test) on the Diabetes dataset.
We compare our BoW routing with the dynamic routing (Sabour et al., 2017) on our architecture.
See rows (1), (2), and (3) in Table 3.
It is obvious that T AB C APS with our proposed BoW routing outperforms that with the dynamic routing (Sabour et al., 2017) in the Diabetes data classification task by ∼ 2 × 10 − 2 in log-loss values.
We also compute the fps of the BoW routing and dynamic routing, and find that BoW routing is nearly 25% faster than the dynamic routing, validating that our routing design benefits tabular learning. [Figure]
Compared with the simple projection (without “entmax”), our sparse projection is witnessed to be beneficial and is compatible with the BoW Routing , which shows that the used “entmax” finds the useful features and excludes the useless ones.
To inspect the capability of our proposed decoder, we conduct a test on missing value imputation, and employ an XGboost classifier to evaluate the effectiveness.
We randomly mask out 20% of the feature values of the instances in the training set of Diabetes , and simply fill the missing values with the intra-class means of the corresponding tabular features.
We train three models on the training set: (i) our T AB C APS with the proposed decoder (using the loss L in Eq.
(8) and MSE), (ii) our T AB C APS with the fully-connection-based decoder of the Vector CapsNet (Sabour et al., 2017) (using L in
Eq.
(8) and MSE), and (iii) an auto-encoder with an auxiliary classifier that takes the hidden states as input (using MSE and cross-entropy loss).
After training, we take the reconstructed feature values provided by these 3 decoders to fill the missing values respectively, and obtain 3 different training sets.
To fairly compare the qualities of the reconstructed values, we train 3 XGboost models with identical hyper-parameters on these 3 training sets, and report their performances on the original test set.
A better performance indicates that the reconstructed values are more trusty.
As reported in Table 3, XGboost trained on the training set provided by our decoder is markedly superior to the other two methods, while the fully-connection-based decoder (Sabour et al., 2017) performs the worst.
Notably, we train another XGboost on the training set in which the masked values remain missing, and find that this XGboost version still obtains 26 .
174 on the test set.
Thus, one can see that our proposed decoder is able to yield significant performance gain by filling the missing values, while the fully-connection-based decoder predicts noisy values.
The effects of capsule layers were tested.
See rows (6), (7), and (8).
The 3-layer and 4-layer T AB C APS obtain slightly better performances on Diabetes compared with the default 2-layer model.
On the
Heart dataset, T AB C APS with 2, 3, 4 capsule layers obtained log-loss values of 0.34047, 0.33992, and 0.33673, respectively.
In general, we consider T AB C APS with different capsule layers perform competitive, and a 2-capsule-layer model is sufficient to attain a good performance.
We also inspected the effects of the “tabular feature extraction” module at the bottom of T AB C APS (see Fig. 1), and the effects of the ensemble of the senior capsules.
Comparing row (9) with (10), and row (11) with (12), it is evident that both of the tabular feature extraction and the senior capsule ensemble are beneficial, but are not the primary causes of superior performances of T AB C APS .
We consider it is the novel architecture makes T AB C APS perform well.
C ONCLUSIONS AND F UTURE W ORK
In this paper, we proposed a novel and simple paradigm for tabular learning, which encapsulated all tabular feature values of an instance into vectorial features as operating units and thus avoids interactions among heterogeneous tabular features.
Consequently, our proposed new CapsNet (called T AB C APS ), with a new primary capsule synthesis approach, a novel sparse projection, and a novel BoW Routing , coped with vectorial features and obtained superior performances on tabular data classification tasks at low computational costs.
Specifically, the proposed BoW Routing performs straightforwardly without iterative clustering processes of previous CapsNets.
This work is a beneficial attempt in the research fields of capsule neural networks, routing algorithms, and tabular learning.
Following previous CapsNets, our proposed method only considered classification tasks on tabular data, and it would be interesting to explore how to use T AB C APS in regression tasks.
Moreover, it is also interesting to design other approach for directly learning instance-level semantics without feature-wise interactions.
Reproducibility Statement.
To ensure the reproducibility, we include the settings of hyperparameter tuning for various methods in Appendix A.1.
Besides, the classification performances measured by accuracy are reported in Appendix A.2 for reference
A A PPENDIX A. H YPERPARAMETER -T UNING S ETTINGS
As mentioned in the main paper, some hyperparameters of methods were turned for datasets by grid search or the HyperOpt library 2 with 50 steps of the Tree-structured Parzen Estimator (TPE), similar to the configurations in (Popov et al., 2019).
The details are listed in the subsections below.
As for XGBoost, CatBoost, FCNN, TabNet, DANets, Vector CapsNet, and our T AB C APS , we used the
HyperOpt hyperparameter search library; as for NODE and Net-DNF, we used grid search (following the settings in their papers).
For FT-T (Gorishniy et al., 2021), we used their official open source.
The hyperparameter search spaces and search algorithms of XGBoost (Chen & Guestrin, 2016),
CatBoost (Prokhorenkova et al., 2018), NODE (Popov et al., 2019), and FCNN (Nair & Hinton, 2010) were set identical as in (Popov et al., 2019).
We set the maximal number of the trees in XGBoost and
CatBoost up to 4096 (instead of 2048 used in NODE (Popov et al., 2019)), since we found thatis not enough for XGBoost and CatBoost to obtain good performances.
We also set the maximal layer number in FCNN up to 9.
Besides, the hyperparameter search settings of Net-DNF (Abutbul et al., 2021) following its original paper.
The architectures of FCNN with lasso regularization or mixup were constructed following the FCNN in (Popov et al., 2019).
For both of DANet-24 (Chen et al., 2022) and TabNet (Arik & Pfister, 2020), we set the hyperparameter search spaces to contain all the possible values provided by the main papers.
A.1. XGB OOST
For easy re-implementation, the hyperparameter variable names followed the sklearn
packagesettings.
• eta (learning rate) : Log-Uniform distribution [ e − 7 , 1]; • max depth : Discrete uniform distribution [2, 10]; • subsample : Uniform distribution [0.5, 1]; • colsample bytree : Uniform distribution [0.5, 1]; • colsample bylevel : Uniform distribution [0.5, 1]; • min child weight : Uniform distribution [ e − 16 , e 5 ]; • alpha : Uniform choice { 0, Log-Uniform distribution [ e − 16 , e 2 ] } ; • lambda : Uniform choice { 0, Log-Uniform distribution [ e − 16 , e 2 ] } ; • gamma : Uniform choice { 0, Log-Uniform distribution [ e − 16 , e 2 ] } .
A.1. C AT B OOST
For easy re-implementation, the hyperparameter variable names followed the sklearn package settings.
• learning rate : Log-Uniform distribution [ e − 5 , 1]; • random strength : Discrete uniform distribution [1, 20]; • one hot max size : Discrete uniform distribution [0, 25]; • l2 leaf reg : Log-Uniform distribution [1, 10]; • bagging temperature : Uniform distribution [0, 1]; • leaf estimation iterations : Discrete uniform distribution [1, 10].
A.1. NODE
• number of layers : { 2, 4, 8 } ; • total number of trees : { 1024, 2048 } ;
• tree depth : { 6, 8 } ; • output dimension of trees : { 2, 3 } ; • learning rate : 10 − 3 .
A.1. N ET -DNF
Following the original setting in (Abutbul et al., 2021)
the learning rate is initialized as 0.05 and is reduced by monitoring the training loss (check its official codes for details).
• number of formulas : { 64, 128, 256, 512, 1024, 2048, 3072 } ; • feature selection beta : { 1.6, 1.3, 1.0, 0.7, 0.4, 0.1 } .
A.1.
[Table]
• λ sparse : { 0.0, 10 − 6 , 10 − 4 , 10 − 3 , 10 − 2 , 0.1 } ; • γ : { 1.0, 1.2, 1.5, 2.0 } ; • batch size B : { 256, 512, 1024, 2048, 4096, 8192, 16384, 32768 } ; • N d and N a : { 8, 16, 24, 32, 64, 128 } ; • N steps : { 3, 4, 5, 6, 7, 8, 9, 10 } ; • B V : { 256, 512, 1024, 2048, 4096 } ; • m B : { 0.6, 0.7, 0.8, 0.9, 0.95, 0.98 } ; • learning rate : { 0.002, 0.01, 0.02, 0.025 } .
A.1. FCNN
• number of layers : Discrete uniform distribution [2, 9]; • output dimensions of full-connections : Discrete uniform distribution { 128, 256, 512, 1024 } ; • learning rate : Uniform distribution [ 10 − 4 , 10 − 2 ]; • dropout rate : Uniform distribution [0, 0.5]; • lasso coefficient (optional) :
Discrete uniform distribution uniform [ 10 − 7 , 10 − 4 ]; • learning rate : Uniform distribution [ 10 − 4 , 0 . 01 ].
A.1. FT-F
• the number of layers : Discrete uniform distribution [1, 4]; • feature embedding size : Discrete uniform distribution [64, 512]; • residual dropout : Uniform distribution [0, 0.2]; • attention dropout : Uniform distribution [0, 0.5]; • FFN dropout : Uniform distribution [0, 0.5]; • FFN factor : Uniform distribution [ 23 , 83 ]; • learning rate : Log-Uniform distribution [ 10 − 5 , 10 − 3 ]; • weight decay : Log-Uniform distribution [ 10 − 6 , 10 − 3 ]; • the number of iterations : 100.
A.1. DAN ET -
The learning rate of DANet-24 is initially set 0.008 and is decayed by 5% in every 20 epochs.
• k 0 : { 5, 8 } ; • d 0 : { 32, 48 } ; • d 1 : { 64, 96 } .
[Table]
learning rate : Uniform distribution [0.02, 0.
[Table]
• primary capsule size : Discrete uniform distribution [64, 128]
[Table]
• the number of primary capsules : Discrete uniform distribution [4, 32]; • senior capsule size : Discrete uniform distribution [4, 32].
A.1. T AB C APS
• learning rate : Uniform distribution [0.02, 0.1]; • the number of senior capsules for a class : Discrete uniform distribution [1, 5]; • primary capsule size : Discrete uniform distribution [64, 128]; • the number of primary capsules : Discrete uniform distribution [4, 32]; • senior capsule size : Discrete uniform distribution [4, 32]; • the number of learnable “words” : Discrete uniform distribution [16, 64].
A. A DDITIONAL CLASSIFICATION P ERFORMANCES ON “A CCURACY ”
Here we report the model performances on the “accuracy” in Table 4.
These classification performances are obtained in 5 runs with regard to random seeds.
One can see that our T AB C APS obtains competitive performances on “accuracy” in comparison with previous work.