TransTab : Learning Transferable Tabular Transformers Across Tables
Anonymous Author(s)
AffiliationAddress email
Abstract
Tabular data (or tables) are the most widely used data format in machine learning (ML).
However, ML models often assume the table structure keeps fixed intraining and testing.
Before ML modeling, heavy data cleaning is required tomerge disparate tables with different columns.
This preprocessing often incurssignificant data waste (e.g., removing unmatched columns and samples).
How tolearn ML models from multiple tables with partially overlapping columns?
How toincrementally update ML models as more columns become available over time?
Can we leverage model pretraining on multiple distinct tables?
How to train an MLmodel which can predict on an unseen table?
To answer all those questions, we propose to relax fixed table structures by introducing a Transferable Tabular Transformer ( TransTab ) for tables.
The goalof TransTab is to convert each sample (a row in the table) to a generalizableembedding vector, and then apply stacked transformers for feature encoding.
Onemethodology insight is combining column description and table cells as the rawinput to a gated transformer model.
The other insight is to introduce supervised andself-supervised pretraining to improve model performance.
We compare TransTabwith multiple baseline methods on diverse benchmark datasets and five oncologyclinical trial datasets.
Overall, TransTab ranks 1.00, 1.00, 1.78 out of 12 methodsin supervised learning, feature incremental learning, and transfer learning scenarios,respectively; and the proposed pretraining leads to 2.3% AUC lift on average overthe supervised learning.
Introduction
Tabular data are ubiquitous in healthcare, engineering, advertising and finance [1, 2, 3, 4].
Theyare often stored in a relational database as tables or spreadsheets.
Table rows represent the datasamples and columns represent the feature variables of diverse data types (e.g., categorical, numerical,binary and textual).
Recent works enhance tabular ML modeling using deep networks [5, 6, 7, 8]or designing self-supervision [2, 9, 10, 11].
Those existing works require the same table structurein training and testing data.
However, there can be multiple tables sharing partially overlappedcolumns in the real world.
Hence, learning across tables is inapplicable.
The traditional remedy isto perform data cleaning by removing non-overlapping columns and mismatched samples beforetraining any ML models, which waste data resources [12, 13, 14].
Therefore, learning across tableswith disparate columns and transferring knowledge across tables are crucial to extending the successof deep learning/pretraining to the tabular domain.
Tables are highly structured yet flexible.
The first step to achieve learning across tables is to rethinkthe basic elements in tabular data modeling.
In computer vision, the basic elements are pixels [15]or patches, [16, 17]; in natural language processing (NLP), the basic elements are words [18] or
Vanilla supervised learning
T cols y T cols y
T cols
T cols y
Fixed-table pretrain+finetune
T cols y T cols y
T cols y …
T2cols
y … (1) Transfer learning across tables
T cols y
T cols
T cols y (2)
Feature incremental learning … (3) Pretrain+finetune
T cols y
T cols
T cols y T cols y
T cols y
T cols (4) Zero-shot inference
Existing works … …
TransTab
Train
Test
Pretrain
Finetune
Pretrain
Finetune
Train
Pretrain
Finetune
Train
Test tokens [19, 20].
In the tabular domain, it is natural to treat cells in each column as independentelements.
Columns are mapped to unique indexes then models take the cell values for trainingand inference.
The premise of this modeling formulation is to keep the same column structure inall the tables.
But tables often have divergent protocols where the nomenclatures of columns andcells differ.
By contrast, our proposed work contextualizes the columns and cells.
For example,previous methods represent a cell valued man under the column gender by 0 referring to the codebook{ man : 0 , woman : 1 } .
Our model converts the tabular input into a sequence input (e.g., gender isman ), which can be modeled with downstream sequence models.
We argue such featurizing protocolis generalizable across tables, thus enabling models to apply to different tables.
In a nutshell, we propose Trans ferable Trans formers for Tab ular analysis ( TransTab ), a versatiletabular learning framework 1 .
TransTab applies to multiple use cases as shown in Fig.
The keycontributions behind TransTab are
• A systematic featurizing pipeline considering both column and cell semantics which is shared as
the fundamental protocol across tables.• V ertical-
P artition C ontrastive L earning ( VPCL ) that enables pretraining on multiple tables and alsoallows finetuning on target datasets.
As shown by Fig.
1, existing works only cover vanilla supervised learning and fixed-table pretrainingdue to the fixed-column assumption.
On the contrary, TransTab relaxes this assumption and appliesto four additional scenarios, which we will elaborate on in §2.1.
Method
In this section, we present the details of TransTab .
Fig.
2 illustrates its workflow including thefollowing key components: 1) The input processor featurizes and embeds arbitrary tabular inputsto token-level embeddings; 2)
The stacked gated transformer layers further encode the token-levelembeddings; 3) Finally, the learning module includes a classifier trained on labeled data and aprojector for contrastive learning.
Next we will present the details of each component.
tokenize & embedding 𝑥 𝑐 cat 𝑥 𝑏 bin 𝑥
𝑢
num
Input processor 𝑥 𝑐 cat 𝑥 𝑏 bin 𝑥
𝑢
num categorical binary numerical
Norm & Align tokenize & embedding cat.
token emb 𝑬 𝑐 tokenize & embedding Element-wise multiply
Element-wise add
Concatenate [CLS] emb.
𝒆 𝑐𝑙𝑠 𝑛 𝑑 bin.
token emb 𝑬 𝑏 𝑛 𝑏 𝑑 num.
token emb 𝑬 𝑢 𝑛 𝑏 𝑑
Norm & Align
Norm & Align 𝑑
Multi-head attention
Gate layer
Linear layer Linear layer
Norm
Gated transformer ×𝐿 blocks 𝑑
Classifier
Projector [
[Figure]
Supervised learning
[Figure]
learning encoded
[Figure]
Input table
[Figure]
Self
[Figure]
supervised
Supervised … …
Learning
2.1 Application scenarios of TransTab
Before presenting our method in details, we first introduce four novel applications scenarios whichare tractable by TransTab , as shown in Fig.
Suppose we aim to predict the treatment efficacy forbreast cancer trials using multiple clinical trial tables, here are several scenarios we often encounter.
S(1) Transfer learning.
We collect data tables from multiple cancer trials for testing the efficacyof the same drug on different patients.
These tables were designed independently with overlappingcolumns.
How do we learn ML models for one trial by leveraging tables from all trials?
S(2) Incremental learning.
Additional columns might be added over time.
For example, additionalfeatures are collected across different trial phases.
How do we update the ML models
using tablesfrom all trial phases?
S(3) Pretraining+Finetuning.
The trial outcome label (e.g., mortality) might not be always availablefrom all table sources.
Can we benefit pretraining on those tables without labels?
How do we finetunethe model on the target table with labels?
S
4) Zero shot inference.
We model the drug efficacy based on our trial records.
The next step is toconduct inference with the model to find patients that can benefit from the drug.
However, patienttables do not share the same columns as trial tables so direct inference is not possible.
Overall, we witness that the assumption of fixed table structure is the obstacle to use ML for variousapplications.
Next we will present TransTab and demonstrate how it addresses these scenarios.
2.2 Input processor for columns and cells
We build the input processor (1) to accept variable-column tables (2) to retain knowledge acrosstabular datasets.
The idea is to convert tabular data (cells in columns) into a sequence of semanticallyencoded tokens.
We utilize the following observation to create the sequence: the column description(e.g., column name) decides the meaning of cells in that column.
For example, if a cell in columnsmoking history has value 1 , it indicates the individual has a smoking history.
Similarly, cell value 60in column weight indicates 60 kg in weight instead of 60 years old.
Motivated by the discussion, we propose to include column names into the tabular modeling.
As a result, TransTab treats any tabulardata as the composition of three elements: text (for categorical & textual cells and column names),continuous values (for numerical cells), and boolean values (for binary cells) .
Fig.
2 illustratesa visual example of how these elements are leveraged to process the four basic types of features:categorical/textual cat , binary bin , and numerical num .
Categorical/Textual feature.
A category or textual feature contains a sequence of text tokens.
Forthe categorical feature cat , we concatenate the column name with the feature value x c , which formsas a sequence of tokens.
This sentence is then tokenized and matched to the token embedding matrixto generate the feature embedding E c ∈ R n c × d where d is the embedding dimension and n c is thenumber of tokens.
Binary feature.
The binary feature bin is usually an assertive description and its value x b ∈ { 0 , 1 } .
If x b = 1 , then bin is tokenized and encoded to the embeddings E b ∈ R n b × d ; if not, it will not beprocessed to the subsequent steps.
This design significantly reduces the computational and memorycost when the inputs have high-dimensional and sparse one-hot features.
Numerical feature.
We do not concatenate column names and values for numerical feature becausethe tokenization-embedding paradigm was notoriously known to be bad at discriminating numbers[21].
Instead, we process them separately.
num is encoded as same as cat and bin to get E u,col ∈ 103 R n u × d .
We then multiply the numerical features with the column embedding to yield the numericalembedding as E u = x u × E u,col , which we identify gets an edge on more complicated numericalembedding techniques empirically.
At last, E c , E u , E b all pass the layer normalization [22] and the same linear layer to be aligned to thesame space, then are concatenated with [cls] embedding to yield E =
˜
E c ⊗ ˜ E u ⊗ ˜ E b ⊗ e
[ cls ] .
As a result, the meaning of all cells are contextualized, i.e., the semantic meaning of one element canvary depending on the context composition.
This formulation benefits the knowledge transfer acrosstables a lot.
For example, previously smoked depicts the same thing as smoking history .
Previousmethods never capture this connection while it is possible for TransTab learn to recognize that 1under both columns are equivalent.
2.3 Gated transformers
The gated tabular transformer is an adaption of the classical transformer in NLP [23].
It consistsof two main components: multi-head self-attention layer and gated feedforward layers.
The inputrepresentation Z l at the l -th layer is first adopted for exploring interactions between features:
[Equation]
where Z
=
E at the first layer; W O ∈ R d × d ; { W Qi , W Ki , W Vi } are weight matrices (in R d × dh ) ofquery, key, value of the i -th head self-attention module.
The multi-head attention output Z l att is further transformed by a token-wise gating layer as g l =σ ( Z l att w G ) , where σ ( · ) is a sigmoid function; g l ∈ [0 , 1] n controls the magnitude of each tokenembedding before Z att goes to the linear projection.
This gates then filters the linear layer output
[Equation]
to obtain the transformer output Z l +1 ∈ R n × d .
This mechanism is learnt to focus on importantfeatures by redistributing the attention on tokens.
The final [cls] embedding z [ cls ] at the L -th layeris used by the classifier for prediction.
2.4 Self-supervised and supervised pretraining of TransTab
The input processor accepts variable-column tables, which opens the door for tabular pretraining onheterogeneous tables.
In detail, TransTab is feasible for self-supervised and supervised pretraining .
Self-supervised VPCL .
Most SSL tabular methods work on the whole fixed set of columns [2, 24, 11],which take high computational costs and are prone to overfitting.
Instead, we take tabular verticalpartitions to build positive and negative samples for CL under the hypothesis that the powerful representation should model view-invariant factors.
In detail, we subset columns as illustratedby Fig.
3 where Self- VPCL is on the top right.
Suppose a sample x i
= { v 1 i , . .
, v Ki } with Kpartitions.
Neighbouring partitions can have overlapping regions which are justified by the percentageof columns of the partition.
Self- VPCL takes partitions from the same sample as the positive andothers as the negative:
[Equation]
where B is the batch size; ψ ( · , · ) is the similarity function.
ψ applies to ˆ z [ cls ] which is the linearprojection of partition v ’s embedding z [ cls ] .
Compared with vanilla CL like SCARF [11], Self- VPCLsignificantly expand the positive and negative sampling for learning more robust and rich embeddings.
What is more, this vertical partition sampling is extremely friendly to column-oriented databases [25]which support the fast querying a subset of columns from giant data warehouses.
For the sake ofcomputational efficiency, when K > 2 , we randomly sample two pieces for computing Eq.
[Figure]
the sample belonging to the same class.
Supervised VPCL .
When we own labeled tabular data for pretraining, one natural idea wouldbe taking task-specific predicting heads for pretraining on vanilla supervised loss, e.g., crossentropy loss.
In finetuning, these heads aredropped and a new head will be added on topof the pretrained encoder.
However, we argueit is suboptimal and may undermine the modeltransferability.
The reason behind is that tabulardatasets vary dramatically in size, task definition,and class distributions.
Pretraining TransTabusing supervised loss inevitably causes the encoder biased to the major tasks and classes.
Moreover, the suitable hyperparameter range isoften distinct across tabular data when applyingsupervised loss.
The same set of hyperparameters can cause overfitting on one dataset andunderfitting on another.
Therefore, it is tricky topick appropriate hyperparameters for pretraining based on vanilla supervised loss.
In this paper, we propose VPCL for pretraining inspired by supervised CL [26] which was provedrobust to noise and hyperparameters.
As illustrated by Fig.
3, we build positive pairs consideringviews from the same class except for only from the same sample:
[Equation]
y =
{ y i } Bi are labels; 1 {·} is indicator function.
VPCL relieves multiple pretraining predictorsrequired to adjust to different datasets.
Moreover, VPCL exposes more feature embeddings to thesupervision by partitioning hence providing more discriminative and generalizable representations.
Experiments
In this section, we aim at answering the following questions by extensive experiments:
• Q1.
How does TransTab perform compared with baselines under the vanilla supervised setting?
• Q2.
How well does TransTab address incremental columns from a stream of data (S(2) in Fig.
1)?• Q3.
How is the impact of TransTab learned from multiple tables (with different columns)
drawnfrom the same domain on its predictive ability (S(1) in Fig. 1)?
• Q4.
Can TransTab be a zero-shot learner when pretrained on tables and infer on a new table (S(4)in Fig. 1)? [Table]
[Table]
• Q5.
Is the proposed vertical partition CL better than vanilla supervised pretraining and self- supervised CL (S(3) in Fig. 1)?
Datasets.
We introduce clinical trial mortality prediction datasets where each includes a distinctgroup of patients and columns 2 .
The data statistics are in Table 1.
Accurately predicting the patientmortality in clinical trials is crucial because it helps identify catastrophic treatment then save patientsfrom harm and improve the clinical trial design.
Considering they are from a similar domain, wecan utilize them to test if TransTab can achieve transfer learning.
Besides, we also include a set ofpublic tabular datasets, the statistics are in Table 7.
Dataset pre-processing.
For all baselines, we represent categorical features by ordinal encoding ifthey need to specify categorical features, otherwise one-hot encoding is used.
Numerical featuresare scaled to [0 , 1] by min-max normalization.
Exceptionally for TransTab , we map the categoricalfeature index to its original description, e.g., mapping class "1" under "gender" to "female".
Model and implementation protocols.
Unless specified otherwise, we keep the settings fixed acrossall experiments.
TransTab uses 2 layers of gated transformers where the embedding dimensionsof numbers and tokens are 128, and the hidden dimension of intermediate dense layers is 256.
Theattention module has 8 heads.
We choose ReLU activations and do not activate dropout.
We train
TransTab using Adam optimizer [27] with learning rate in { 2 e-5 , 5 e-5 , 1 e-4 } and no weight decay;batch size is in { 16 , 64 , 128 } .
We set a maximum self-supervised pretraining epochs of 50 andsupervised training epochs of 100.
A patience of 10 is kept for supervised training for early stopping.
Experiments were conducted with one RTX3070 GPU, i7-10700 CPU, and 16GB RAM. [Table]
Baselines.
We include the following baselines for comparison: Logistic regression (LR) ; XGBoost[28]; Multilayer perceptron (MLP) ; SeLU MLP (SNN) [29]; TabNet [30]; DCN [1]; AutoInt [31]; TabTransformer [5]; FT-Transformer [32]; VIME [2]; SCARF [11].
We provide the baseline architecturesand implementations in Appendix B.
3.1 Q1. Supervised learning
Results of supervised learning on clinical trial mortality prediction datasets are summarized by Table 2.
We see that our method outperforms baselines on all.
From the view of method ranks, we surprisinglyidentify that LR wins over half of baseline methods.
Except for TransTab , FT-Transformer is theonly model that shows significant superiority over LR, which illustrates the potential of transformersfor tabular modeling.
Additional results on public datasets are available in Table 8 where we witnessthat our method is comparable to the state-of-the-art baseline tabular models.
We also discover thebaselines drawn from the CTR prediction literature (DCN and AutoInt) turn out the be competitive intabular modeling.
3.2 Q2. Feature incremental learning
For previous tabular models, we should either drop new features or drop old data when confrontingfeature incremental learning.
By contrast, TransTab is able to continually learn from new datawith incremental features.
We split each dataset into three subsets to mimic the incremental featurescenario, as illustrated by (5) in Fig.
Baseline models can be trained by set1+2+3 with features inset1 or only by set3; TransTab can learn from set1+set2+set3.
Table 3 shows the results where wefind our method outperforms baselines by a great margin.
It demonstrates that TransTab makes thebest of incremental features to learn better.
Similar observations appear in public datasets, shown by Table 9.
3.3 Q3. Transfer learning
We further test if TransTab is able to transfer knowledge across tables.
Results are in Table 4.
Wesplit each dataset into two subsets with 50% overlaps of their columns.
Baselines are trained andtested on set1 (only label-supervision) or set2 separately.
For our method we pretrain it on set1 thenfinetune it on set2 and report its performance on set2, and vice versa.
We observe that TransTab canbenefit from knowledge transfer across tables to reach superior performances.
Similar observationsare made on public datasets shown by Table 10.
3.4 Q4. Zero-shot learning
Although there are numerous papers on zero-shot learning (ZSL) in CV and NLP [33, 34, 35], wenotice that ZSL was hardly mentioned in tabular domain. In this experiment, we refer to the ZSL
[Table]
[Table]
scenario mentioned by S(4) of Fig.
1 where we split the raw table into three equal-size subsets.
Three subsets have distinct columns.
For the zero-shot setting, the model learns from set1+set2 andis tested on set3 without further training.
In this scenario, the model needs to leverage the learnedknowledge from set1 and set2 to support the inference on a new table set3.
Besides, we design twobaselines for comparison: supervised where the model learns from set3 and predicts on set3 andtransfer where the model learns from set1+set2 and continues to be finetuned on set3.
Results arein Table 5.
We surprisingly find the ZSL model gets better performance than the supervised one onaverage.
It boils down to that (1) ZSL TransTab succeeds to retain the learned knowledge fromset1+set2 for predicting on a new table (set3) and (2) ZSL can benefit from more data (set1+set2)than the supervised (set3 only).
Meanwhile, the transfer model takes the advantage of set1+set2 andis adapted for set3 by finetuning, hence reaches the best performance.
Similarly, we witness that
TransTab is able to make zero-shot predictions on public datasets as in Table 11.
3.5 Q5. Supervised and self-supervised pretraining
We take experiments to compare the proposed VPCL with the vanilla transfer learning strategy, asin Table 6.
We observe that the vanilla strategy harms the performance on two datasets while VPCLalways brings positive effect for finetuning.
Besides, we conduct experiments on varying the numberof partitions and show the average AUROC on all five datasets, shown by Fig.
We specify that
VPCL demonstrates an advantage over self- VPCL when we increase the partition numbers.
We also explore if pretraining works on public datasets.
Results in Table 12 somewhat match ourexpectations that pretraining on unrelated tabular data usually yields few benefits for finetuningbecause these tables define totally different columns and targeted tasks.
We also show the ablationon the number of partitions by Fig.
5 where VPCL consistently outperforms the Supervised baseline.
Nevertheless, it is still worth investigating the table phenotypes to aggregate tables which are morelikely to benefit from each other by transfer learning.
Related Works
Tabular Prediction.
To enhance tabular predictions, numerous recent works try to design newalgorithms [28, 36, 37, 30, 38, 32, 5, 10, 7, 39, 40, 41, 42, 43, 44, 45].
However, it was argued thatboosting algorithms and MLPs are still the competitive choices for tabular data modeling, especially [Table] when the sample size is small [32, 46, 39, 47].
To alleviate label scarcity issue, SSL pretraining onunlabeled tabular data was introduced [2, 24, 10, 9, 11].
Nonetheless, none of them is transferableacross tables then is able to extend the success of pretraining to the tabular domain.
For practicaltabular predictions, the common case is that we own a lot of labeled samples collected with diverseprotocols hence heavy preprocessing is needed to align them by either dropping many samples ormany features.
By contrast, TransTab accepts variable-column tables and therefore can learn fromdifferent tables at scale and transfer to the target task.
Also, it can support diverse tabular predictiontasks as depicted in Fig.
1, which cannot be done by off-the-shelf tabular methods.
Transfer learning.
Transfer learning (TL) has long been a popular research field since the proposalof ImageNet [48], which gives rise to splendid works on utilizing supervised pretraining on a largegeneral database and finetune on a small downstream task [49, 50, 51, 52, 53].
TL is also fast-growingin NLP beginning at BERT [20], which often leverages web-scale unlabeled texts for self-supervisedpretraining and then applies to specific tasks [34, 54, 55, 56, 57].
However, few work was on TLin tabular predictions.
As mentioned in §1, TransTab paves the way for effective tabular TL byestablishing a feature processing protocol that applies for most table inputs, such that it sharesknowledge across tables.
Self-supervised learning & contrastive learning.
SSL uses unlabeled data with pretext tasks to learnuseful representations and most of them are in CV and NLP [20, 17, 15, 16, 58, 23, 59, 60, 61, 62, 63].
Recent SSL tabular models can be classified into reconstruction and contrastive based methods:
TabNet [30] and VIME [2] try to recover the corrupted inputs with auto-encoding loss; SCARF [11]takes a SimCLR-like [64] contrastive loss between the sample and its corrupted version; SubTab[9] takes a combination of both.
Nevertheless, all fail to learn transferable models across tablessuch that cannot benefit from pretraining with scale.
Contrastive learning can also be applied tosupervised learning by leveraging class labels to build positive samples [26].
Our work extends it toto the tabular domain, which we prove works better than vanilla supervised pretraining.
The verticalpartition sampling also enjoys high query speed from large databases which are often column-oriented[25].
Another line of research takes table pretraining based on masked language modeling for tablesemantic parsing [65, 66, 67, 68].
But it remains elusive if these methods can bring benefits to tabularpredictive tasks.
Conclusion
This paper proposes TransTab that accepts variable-column inputs.
By the proposed vertical partitioncontrastive learning, it can benefit from supervised pretraining from multiple tabular datasets withlow memory cost.
We envision it to be the basis of tabular foundation models and widely used totabular-related applications in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See § 3.5 and Appendix §A.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix §A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
This paperdoes not include theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
This paper does notinclude theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary materials.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] For our methods please see Model and Implementation Protocols of§3; for the compared baselines please see Appendix §B;
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Table 13.
(b) Did you mention the license of the assets?
[Yes] Licenses are available referring to the provided links.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See Appendix §C.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Appendix §C.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes] See Appendix §C.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
This paper does not use crowdsourcing.
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
This paper does not use crowdsourcing.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
This paper does not use crowdsourcing.
A Broader impact of this work
Tabular data is the most common data format used in practical data analysis including healthcare,finance, business, advertising, manufacturing, etc.
Regardless of its importance, much more effortwas made to play deep learning with other domains like vision, language, and audio.
As a result,non-deep algorithms especially tree-based still dominate tabular data analysis.
This paper opens thedoor to leverage the power of transfer learning in the tabular domain because TransTab is capable ofdealing with variable-column input tables.
This property also alleviates the workload required fordata preprocessing because TransTab permits missing features and make good predictions based onthe remaining features.
This advance is expected to bring tremendous savings of time and moneyfrom the data engineering which often takes up 80% of efforts for data science projects [69].
Italso sheds light on developing foundation models for the tabular domain because of the success ofpretraining on scale.
The potential negative effect would be that it requires more effort on solving data privacy issuesbecause TransTab works better when features are represented by descriptive texts compared withdiscretized indices.
However, this can be alleviated by mapping features to machine-readable tokensin advance by a private codebook.
Besides, TransTab needs more resources than simple modelslike MLP and Trees.
The main cause is the use of full attention in multihead attention modulesand the tokenization in the featurizing process.
The former problem can be alleviated by replacingtransformers with MLP-based blocks like gated attention units.
The latter can be circumvented bypre-tokenization before the model training.
B Baseline architecture and implementation
In this section, we introduce the implementation details of baselines in experiments.
• LR: Use the default setting of the package scikit-learn 3 except the max iterations is set 1000.• XGBoost:
We set the maximum number of estimators in { 50 , 100 , 500 } and the max depth in{ 4 , 6 , 8 } .
Implemented based on the XGBoost package 4 .
• MLP & SNN:
We keep the same architecture for both except for their activations: three denselayers with hidden dimensions 256, 256, 1; dropout with rate of 0.1 is used.
They are trained withbatch size ∈ { 16 , 32 , 64 , 128 } , learning rate ∈ {5e-5, 1e-4, 5e-4, 1e-3}, and early stopping patienceof 5 with 100 maximum epochs.• TabNet:
Use the official implementation with the default recommended parameters 5 .
Trainedwith batch size ∈ { 16 , 32 , 64 , 128 } , learning rate ∈ {1e-4, 1e-3, 2e-2}, n a , n d ∈ { 8 , 16 , 64 , 128 } ,γ ∈ { 1 . 3 , 1 . 5 , 1 . 8 } , categorical embedding dimension ∈ { 1 , 8 , 16 } and early stopping patience of5 with 100 maximum epochs. [Table] • DCN: Use the implementation by Deep-CTR 6 .
The number of cross is 2; dropout rate for feedforward component is 0.1; MLP part has two dense layers of dimension 256, 128; Trained withbatch size ∈ { 16 , 32 , 64 , 128 } , learning rate ∈ {5e-5, 1e-4, 1e-3}, and early stopping patience of10 in 100 maximum epochs.• AutoInt: Use the implementation by Deep-CTR.
Attention layer number is set 2; Attention headnumber is set 2; MLP part has two dense layers of dimension 256, 128; dropout deactivated;Trained with batch size ∈ { 16 , 32 , 64 , 128 } , learning rate ∈ {5e-5, 1e-4, 1e-3}, and early stoppingpatience of 10 in 100 maximum epochs.• TabTransformer:
Use the official implementation 7 .
Feed-forward component has 128 dimension;2 transformer layers are used; The number of heads of attention is ∈ { 2 , 4 , 8 } ; Dropout rateis 0.1; ReLU activation is used; Trained with batch size ∈ { 16 , 32 , 64 , 128 } , learning rate ∈{5e-5,1e-4,1e-3}, and early stopping patience of 10 in 100 maximum epochs.
• FT-Transformer:
Use the official implementation 8 .
Feed-forward component has 128 dimension;
transformer layers are used; The number of heads of attention is ∈ { 2 , 4 , 8 } ; Dropout rate is 0.1; 3 sklearn.linear_model.LogisticRegressionxgboost.sklearn 5 https://github.com/dreamquark-ai/tabnethttps://github.com/shenweichen/DeepCTR-Torchhttps://github.com/lucidrains/tab-transformer-pytorch https://github.com/Yura52/rtdl
[Table] [Table]
DCN 0.
AutoInt 0.
TabTrans 0.
FT-Trans 0.
VIME 0.
SCARF 0.
TransTab 0.
ReLU activation is used; Trained with batch size ∈ { 16 , 32 , 64 } , learning rate ∈ {5e-5,1e-4,1e-3},
and early stopping patience of 10 in 100 maximum epochs
• VIME: We reproduce it by PyTorch [70] based on the original official implementation 9 where its
encoders, mask estimators, decoders are all one dense layer with the hidden dimension same as the
input features.
During the pretraining phase, we train the model on all training data taking maskrate 0.3, batch size 128, learning rate 1e-4, and 10 epochs; during the fine-tuning phase, we add aclassifier after the encoder with three dense layers of 100 dimension and ReLU activations.
Trainedwith batch size ∈ { 16 , 32 , 64 , 128 } , learning rate {5e-5,1e-4,1e-3}, and early stopping patience of10 in 100 maximum epochs.• SCARF: Since no official code is found, we reproduce it by PyTorch based on the descriptionsin the original paper.
A 4-layer encoder and a 2-layer decoder with ReLU activations are used.
Hidden dimensions of their intermediate layers are all 256.
During the pretraining phase, we trainthe model on all trainign data taking mask rate 0.5, InfoNCE loss [71] with learning rate 1e-3,batch size 128, and 10 epochs; during the fine-tuning phase, we add a classifier after the encoderwith two dense layers with 256 hidden dimensions.
Trained with batch size ∈ { 16 , 32 , 64 , 128 } ,learning rate ∈ {5e-5,1e-4,1e-3}, and early stopping patience of 10 in 100 maximum epochs.
C Preprocessing of clinical trial datasets
The raw real-world de-identified patient-level clinical records are obtained from Project Data Sphere.
All clinical trial data used in this paper are available under registration for the data platform.
For
[Table] [Table]
each trial, we manage to extract patient’s baseline information as the features, including demographicinformation, medical history, medication history, lab test, vital signs, adverse events
[Table]
etc.
We drawthe target labels from the survival analysis section where censoring is considered as "alive" and otherevents are tagged "mortality" so as to transform the datasets into binary prediction tasks.
[Table]
[Table]
[Table]
[Figure]
[Figure]
[Table]