Paper: Unlocking the Transferability of Tokens in Deep Models for Tabular Data (ICLR-2024)

Reviewer comment (Novelty):
Significance: Lin et al. 2023 [1] demonstrated that the output layer of pre-trained text encoders do encode similarities or structure in tokens. This seems to contradict the authors claim and Figure 4 that vanilla training does not contain structure of the features? Is the lack of feature token transferability specific to tabular datasets? If so, is this solving a narrow or a domain specific problem, or would it generalize to any language model that uses a tokenizer?

Author response:
**Q1:** Pre-trained text encoders are able to encode similarities or structures in tokens. Is the lack of feature token transferability specific to tabular datasets?

**A1:** 

BERT Text Encoder and CLIP Text Encoder, just as mentioned in the paper [1] cited by the reviewer, are pre-trained on vast corpora of data and have large-scale models. Through extensive data and pre-training methods for language model, the pre-trained text encoders possess semantic representation capabilities. However, due to the diverse feature types and different feature dimensions across different tabular tasks, pre-training for tabular data is typically conducted on a single task [A] or on multiple tasks without sharing tokenizers [B]. Because it is currently not feasible to simultaneously train a feature tokenizer on a large volume of heterogeneous tabular datasets, the pre-training tokenizer lacks token transferability.

So the observation that the learned tabular tokens with poor semantics does not contradict with the semantic tokens of large NLP models. TabToken is solving a problem for tabular data where the learned tokens cannot encode similairities or structures, which makes the encoder of deep tabular models cannot be transfered as those in image processing and natural language processing domains.  In resource-constrained scenarios, our approach may generalize to a language model, assisting the text encoder in obtaining more semantic information.

Reviewer comment (Experiments):
Incremental performance: Table 1 shows that TabToken only leads to a marginal improvement (< 0.2 accuracy and < 3 RMSE), some of the baselines are non deep learning methods. Figure 6, also shows marginal improvement of TabToken compared to other methods with increasing feature ratios. Table 2 also shows incremental improvement or even a decrease in performance when combining existing models with CTR. Is this why the improvement or deprovement with CTR was not highlighted using bold numbers in table 2?

Author response:
**Q2:** Some of the baselines are non-deep learning methods. The improvement is not enough.

**A2:** 

Unlike the success observed in text and image datasets, deep models still struggle to outperform non-deep models on tabular data. Tree-based models remain competitive on tabular data even without accounting for their superior speed [C]. There is still no universally superior solution among GBDT and deep models [D]. Besides, in the experiments for XTab [B], with hyperparameter optimization, XTab ranks lower than XGBoost. Therefore, our comparison is between the currently strong baselines in both non-deep and deep methods. In our setting, TabToken ranks high across the majority of datasets. In Table 2, due to the diversity of datasets, it is challenging for CTR to achieve universal improvements. However, we still achieved improvements on the majority of datasets. 

One of the main contributions of our approach lies in enhancing the transferability of deep models. As shown in Table 12, when we transfer the pre-trained Transformer and fine-tune it, the pre-training model demonstrates a significantly lower transfer effect compared to TabToken. Therefore, we have made improvements in the transferability of deep tabular models.

Reviewer comment (Experiments):
Difficulty of datasets used: The number of features in the datasets span between 8 to 54 in table 3. With a small dataset scale, non deep learning methods actually perform better. Could the paper be a case of using an over-engineered solution? More importantly, the benefits of using TabToken to improve deep learning algorithms becomes questionable since the dataset complexity is not at scale to employ deep learning methods. Perhaps pre-training or finetuning only the tokenizer might be relevant and the encoded features can be passed to a non deep learning classifier such as SVM? This could alleviate the complexities of training deep learning classifiers and focus the problem of learning good feature representations.

Author response:
**Q3:** Non-deep learning methods perform better on small scale datasets. The dataset complexity is not at scale.

**A3:**

Firstly, the inferior performance of deep learning compared to non-deep models is not primarily due to small-scale issues. In comparisons on large-size datasets, this paper [C] found that, although the advantage of tree models diminishes a little, a gap still exists between deep models and tree models. (The Jannis dataset in our experiments falls into the category of large-size datasets as classified in this paper.) The challenges that hinder deep models from outperforming non-deep models on tabular data stem from various factors [C]: neural networks are inclined towards overly smooth solutions, MLP-like neural networks are more affected by uninformative features, and data lack invariance to rotation.

Secondly, applying deep learning methods to small-scale tabular datasets has received attention in recent literature, such as TabPFN [E], where the deep learning methods are expected to possess the model prior and be transferred to downstream tasks more effectively. In our experiments, we show our TabToken can perform well on small-scale datasets, better than TabPFN.

Thanks for the suggestions. We conducted experiments on two larger and more complex datasets. We collect scene recognition dataset (300 features, binary classification) and sylva agnostic dataset (217 features, binary classification) from OpenML. We use the same setting as the experiments in Table 1. The experimental results indicate that non-deep methods remain competitive on complex datasets, showing significant advantages in scene recognition dataset. Moreover, TabToken continues to be an advantageous solution among deep models on complex datasets.

| Acc (%) | XGBoost   | XTab  | TabRet | TabToken  |
| ------- | --------- | ----- | ------ | --------- |
| scene   | **56.24** | 42.97 | 39.58  | 53.87     |
| sylva   | 68.15     | 24.59 | 35.74  | **72.92** |

Reviewer comment (Reproducibility):
Will the algorithm be released as a pytorch or tensorflow module that can be incorporated into other models?

Author response:
**Q5:** Will the algorithm be released as a pytorch or tensorflow module that can be incorporated into other models?

**A5:** 

TabToken implements the idea through regularizing the learned tokens. The corresponding transferability of the model could be improved. It helps token-based methods in general, especially those deep tabular models.

Reviewer comment (Experiments):
They do not characterize the heterogeneity of the datasets they evaluate on -- hence, I do not have a sense for how hard it actually is to transfer between feature sets.

Author response:
**Q1:** Characterize the heterogeneity of the datasets. 

**A1:** 

The heterogeneity of the dataset is reflected in the variations of the feature space. Specifically, there are features in the pre-training dataset that are absent in the downstream tasks, and vice versa, downstream tasks include features that were not encountered during the pre-training phase. The heterogeneity of the dataset can also be reflected in the distinct domains across datasets, as exemplified by the pre-training dataset BRFSS and the downstream datasets Diabetes & Stroke in Table 9. We construct a pre-training task on BRFSS for predicting HIV test experience. The task of Diabetes dataset is to predict whether a subject has diabetes. The Stroke dataset records clinical events and the task is to predict whether a subject is likely to get a stroke. 

**Q2:** How hard it actually is to transfer between feature sets.

**A2:**

Traditional tabular models, such as XGBoost, face challenges in adapting to changes in feature inputs during transfer. Each tabular feature has a strong correspondence with model parameters, making it hard to directly transfer pre-trained deep learning models when encountering unseen features. As demonstrated in our ablation study in Table 12, when we fix a certain part of the pre-trained Transformer and fine-tune it, the pre-training model exhibits a lower transfer effect compared to TabToken. The pre-trained deep model faces challenges in achieving the desired transfer effect between feature sets. The scarcity of available samples for fine-tuning on new datasets further complicates the knowledge transfer process.

Reviewer comment (Novelty):
The contrastive loss with the label has limited novelty.

Author response:
Previous transfer methods have focused on how to leverage large language models [A, B] or enhance the transferability of the top-layer model [C, D]. To the best of our knowledge, we are the first to focus on unlocking the transferability of feature tokens in the realm of tabular data transfer learning. We have unveiled a phenomenon starkly distinct from that observed in other domains: tokens learned by tabular data deep models lack semantic understanding. We devised the "Contrastive Token Regularization" (CTR), incorporating token pre-training and token fine-tuning tailored for tabular data.

Although the general form of CTR is similar to the commonly used contrastive loss, the specific CTR is designed for deep tabular model. We start by converting a set of feature tokens to instance tokens through token combination. Then, we use token regularization to pull instance tokens towards their class center. This process enables contrastive loss to regularize the feature tokens effectively. Also, we have explored different forms of contrastive loss in Table 7.

Moreover, we emphsize we can improve the semantic of the token through applying CTR over those tokens *before* the transformations in the deep tabular model, as we demonstrated in Figure 4 and Figure 5.

Furthermore, CTR is just one component of TabToken, and the novelty of our approach also comes from the whole pre-training and fine-tuning method.

Reviewer comment (Novelty):
The section on related works should be integrated into the main article, as it is difficult to discern the specific improvements in comparison to previous methods.

Author response:
Thanks for the suggestions. 

Although we discussed the relationship between our TabToken with previous methods such as Transferring Tabular Models across Feature Spaces, we chose to place the related work section at the beginning of the appendix due to the substantial content of the related work and constraints on the length of the main text. 

We will move the related work to the main text.

Reviewer comment (Experiments):
While the authors experimented with diverse domains of datasets, both the pretraining and finetuning datasets for each experiment originate from the same dataset. It remains uncertain whether the proposed method can be generalized across domains.

Author response:
In the appendix, we have explored the OOD generlization ability of TabToken in Table 9, where TabToken was pre-trained on BRFSS and fine-tuned on Diabetes & Stroke. BRFSS is the Behavioral Risk Factor Surveillance System dataset with more than 340K samples and more than 300 features from Kaggle. We construct a pre-training task on BRFSS for predicting HIV test experience. The task of Diabetes dataset is to predict whether a subject has diabetes. The Stroke dataset records clinical events and the task is to predict whether a subject is likely to get a stroke. 

Although these three datasets belong to different domains, they all fall under the category of health diagnosis. Approximately half of the features in the downstream datasets are present in the pre-training dataset, allowing us to apply TabToken. We conducted pre-training in the domain of health behaviours and transferred knowledge to domains for Diabetes or Stroke. The results (as shown in Table 9) indicate that TabToken outperforms other baselines in this natural cross-domain application with overlapping features. TabToken can be generalized across domains.

Due to concerns of data privacy and security in healthcare and financial applications, access to lots of cross-domain datasets with overlapping features is typically restricted. We follow the dataset splitting in [E,F,G,H,I] to construct pre-training and fine-tuning tasks, where the resources are limited and it is challenging to obtain all features at once.

[A] Tabllm: Few-shot classification of tabular data with large language models. AISTATS, 2023.

[B] Transtab: Learning transferable tabular transformers across tables. NeurIPS, 2022.

[C] Xtab: Cross-table pretraining for tabular transformers. ICML, 2023.

[D] TabRet: Pre-training Transformer-based Tabular Models for Unseen Columns. ICLR workshop, 2023.

[E] Heterogeneous ensemble for feature drifts in data streams. PAKDD, 2012.

[F] One-pass learning with incremental and decremental features. IEEE T-PAMI, 2018.

[G] Online Learning from Data Streams with Varying Feature Spaces. AAAI, 2019.

[H] Learning with feature and distribution evolvablestreams. ICML, 2020.

[I] Heterogeneous few-shot model rectification with semantic mapping. IEEE T-PAMI, 2021.

Reviewer comment (Experiments):
The proposed method necessitates annotated labels for learning semantic tokens, limiting its application to supervised training. A self-supervised pretraining approach without annotations could be more appealing.

Author response:
Firstly, during the upstream task, TabToken needs to train a predictive model. Based on this model, we complete the supervised upstream task while simultaneously enhancing model's transferability. It is different from pre-training a module specifically for the downstream task. Both the pre-training and downstream tasks can leverage our pre-trained model. SCARF [J], a baseline in the main text, is based on a self-supervised pre-training method. In our experimental results, SCARF struggles to demonstrate the benefits of self-supervision. 

Secondly, the relationship between features and labels is a crucial source of semantics. In response to the reviewer's suggestion, we follow SCARF to adopt a self-supervised approach. For each mini-batch of examples from the unlabeled training data, we generate a corrupted version $x'$ for each example $x$. We uniformly sample some fraction (corrupted ratio) of the features and replace each of those features with a random draw from that feature’s empirical marginal distribution. (Normal distribution for numerical features, probabilities for each categorical feature) . We then pass both $x'$ and $x$ through the tokenizer and average their respective outputs. Finally, we L2-normalize the outputs and compute the InfoNCE loss instead of our CTR. We set the corruption ratio (cr) to 0.1, 0.2, and 0.3. Experimental results on four datasets indicate that TabToken with this self-supervised loss struggles to confer transferability to the model on the majority of datasets. We use the same setting as the experiments in Table 1. The results are as follows:

| Acc (%)                       | eye       | jannis    | cardio    | htru      |
| ----------------------------- | --------- | --------- | --------- | --------- |
| SCARF                         | 33.71     | 35.27     | 55.47     | 81.31     |
| TabToken (self-sup, cr = 0.1) | 38.10     | 31.46     | **62.40** | 82.35     |
| TabToken (self-sup, cr = 0.2) | 38.35     | 34.11     | 62.07     | 83.30     |
| TabToken (self-sup, cr = 0.3) | 38.22     | 35.59     | 61.84     | 83.55     |
| TabToken (CTR)                | **39.28** | **36.87** | 62.29     | **84.59** |

Thanks for the suggestion. We will include the relevant experiments in our paper.



[J] Scarf: Self-supervised contrastive learning using random feature corruption. ICLR, 2022.