T AB PFN: A T RANSFORMER T HAT S OLVES S MALL T ABULAR C LASSIFICATION P ROBLEMS IN A S ECOND
Anonymous authors
Paper under double-blind review
A BSTRACT
We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second , needs no hyperparameter tuning and is competitive with state-of-the-art classification methods.
TabPFN is fully entailed in the weights of our network, which accepts training and test samples as a set-valued input and yields predictions for the entire test set in a single forward pass.
TabPFN is a Prior-Data Fitted Network (PFN) and is trained offline once, to approximate Bayesian inference on synthetic datasets drawn from our prior.
This prior incorporates ideas from causal reasoning:
It entails a large space of structural causal models with a preference for simple structures.
On 30 small datasets from the OpenML-CC18 suite, we show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to 70 × speedup.
This increases to a 3 200 × speedup when a GPU is available.
We provide all our code, the trained TabPFN, an interactive browser demo and a Colab notebook at https://github.com/tabpfn-anonym/TabPFNAnonym .
I NTRODUCTION
Tabular data, comprised of categorical and numerical features, has long been overlooked by deep learning research, despite being the most common data type in real-world machine learning (ML) applications (Chui et al., 2018).
While deep learning methods excel on many ML applications, tabular data classification problems are still dominated by Gradient-Boosted Decision Trees (GBDT; Friedman, 2001), largely due to their short training time and robustness (Shwartz-Ziv and Armon, 2022).
We propose a radical change to how tabular classification is done.
We do not fit a new model from scratch to the training portion of a new dataset.
Instead, we replace this step by performing a single forward pass with a large Transformer that has been pre-trained to solve artificially generated classification tasks from a tabular dataset prior.
Our method builds on Prior-Data Fitted Networks (PFNs; Müller et al., 2022; see Section 2), which learn the training and prediction algorithm itself.
PFNs approximate Bayesian inference given any prior one can sample from and approximate the posterior predictive distribution (PPD) directly.
While inductive biases in NNs and GBDTs depend on them being efficient to implement (e.g., through
L 2 regularization, dropout (Srivastava et al., 2014) or limited tree-depth), in PFNs, one can simply design a dataset-generating algorithm that encodes the desired prior.
This fundamentally changes the way we can design learning algorithms.
We design a prior (see Section 3) based on Bayesian Neural Networks (BNNs; Neal 1996; Gal 2016) and Structural Causal Models (SCMs; Pearl 2009; Peters et al. 2017) to model complex feature dependencies and potential causal mechanisms underlying tabular data.
Our prior also takes ideas from Occam's razor: explanations based on simpler SCMs and BNNs (with fewer parameters) have a higher likelihood.
Our prior is defined via parametric distributions, e.g., a log-scaled uniform distribution for the average number of nodes in data-generating SCMs.
The resulting PPD implicitly models uncertainty over all possible data-generating mechanisms, weighting them by their likelihood given the data and their prior probability.
Thus, the PPD corresponds to an infinitely large ensemble of data-generating mechanisms, i.e., instantiations of SCMs and BNNs.
We learn to approximate this complex PPD in a single forward-pass, requiring no cross-validation or model selection.
Our key contribution is to introduce the TabPFN (see Section 4), a single Transformer that has been pre-trained to approximate probabilistic inference for the novel prior above in a single forward pass, and has thus learned to solve novel small tabular classification tasks ( ≤ 1 000 training examples,features and 10 classes) in less than a second while yielding state-of-the-art performance.
To substantiate this claim, we qualitatively and quantitatively analyze the behavior and performance of our TabPFN on different tasks and compare it to previous approaches for tabular classification on 30 small datasets (see Section 5).
Quantitatively, the TabPFN yields much better performance than any individual “base-level” classification algorithm, such as gradient-boosting via XGBoost (Chen and Guestrin, 2016), LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018), and in less than a second yields performance competitive to what the best available AutoML frameworks (Erickson et al., 2020; Feurer et al., 2021) achieve in 5 to 60 minutes.
Our in-depth qualitative analysis shows the TabPFN's predictions to be smooth and intuitive.
Yet, its errors are quite uncorrelated to the errors of existing approaches, allowing additional performance improvements by ensembling.
We expect the revolutionary character of our claims to be met with initial scepticism and thus opensource all our code and the pre-trained TabPFN for scrutinization by the community, along with a scikit-learn-like interface, a Colab notebook and two online demos.
Please see Section 8 for the links.
B ACKGROUND ON P RIOR -D ATA F ITTED N ETWORKS
First, we summarize how PFNs work; we refer to Müller et al.
for the full details.
The Posterior Predictive Distribution for Supervised Learning In the Bayesian framework for supervised learning, the prior defines a space of hypotheses Φ on the relationship of a set of inputs x to the output labels y .
Each hypothesis ϕ ∈ Φ can be seen as a mechanism that generates a data distribution from which we can draw samples forming a dataset.
For example, given a prior based on structural causal models, Φ is the space of structural causal models, a hypothesis ϕ is one specific
SCM, and a dataset comprises samples generated through the SCM.
In practice, a dataset comprises training data with observed labels and test data where labels are missing or held out to assess predictive performance.
The PPD for a test sample x test specifies the distribution of its label p ( ·| x test , D train ) , which is conditioned on the set of training samples D train :
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } .
The PPD can be obtained by integration over the space of hypotheses Φ , where the weight of a hypothesis ϕ ∈ Φ is determined by its prior probability p ( ϕ ) and the likelihood p ( D | ϕ ) of the data D given ϕ :
[Equation]
Synthetic Prior-fitting Prior-fitting is the training of a PFN to approximate the PPD and thus do
Bayesian prediction.
We implement it with a prior which is specified by a prior sampling scheme of the form p ( D ) =
E ϕ ∼ p
ϕ ) [ p ( D | ϕ )] , which first samples hypotheses (generating mechanisms) with ϕ ∼ p ( ϕ ) and then synthetic datasets with D ∼ p
( D | ϕ ) .
We repeatedly sample such synthetic datasets D :=
( x i , y i ) i ∈{ 1 ,...,n } and optimize the PFN's parameters θ to make predictions for
D test ⊂ D , conditioned on the rest of the dataset D train =
D \ D test .
The loss of the PFN training thus is the cross-entropy on held-out examples of synthetic datasets.
For a single test point { ( x test , y test ) }
=
D test , the training loss can be written as
[Equation]
As shown by Müller et al. (2022), minimizing this loss approximates Bayesian prediction.
We visualize this in Figure 1a and detail the full training setup in Algorithm 1 in the appendix.
Crucially, this synthetic prior-fitting phase is performed only once for a given prior p ( D ) as part of algorithm development, learning to do real-world inference on new datasets.
Real-World Inference During inference
, the trained model is applied to unseen real-world datasets.
For a novel dataset with training samples D train and test features x test , feeding ⟨ D train , x test ⟩ as an input to the model trained above yields the PPD q θ ( y | x test , D train ) in a single forward-pass.
The
PPD class probabilities are then used as predictions for our real-world task.
Thus, PFNs perform training and prediction in one step (similar to prediction with Gaussian Processes) and do not use gradient-based learning on data seen at inference time.
[Figure]
Architecture PFNs rely on a Transformer (Vaswani et al., 2017) that encodes each feature vector and label as a token, allowing token representations to attend to each other, as depicted in Figure 1b.
They accept a variable length training set D train of feature and label vectors (treated as a set-valued input to exploit permutation invariance) as well as a variable length query set of feature vectors x test =
{ x ( test, 1) , . . .
, x ( test,m ) } and return estimates of the PPD for each query.
Tabular Data Amongst other experiments, Müller et al.
(2022) demonstrated PPD approximation with PFNs on binary classification for very small, balanced tabular datasets with 30 training examples.
Here, we substantially improve performance and scale up to realistically-sized small datasets with up to 2 000 data points and up to 10 imbalanced classes.
We discuss detailed changes in Appendix C.2.
. A P RIOR FOR T ABULAR D ATA
The performance of our method crucially depends on the specification of a suitable prior, as the PFN approximates the PPD for this prior.
Section 3.1 outlines a fundamental technique for our prior: we use distributions instead of point-estimates for almost all of our prior’s hyperparameters.
Section 3.
motivates simplicity in our prior, while Sections 3.3 and 3.4 describe how we use structural causal models (SCMs) and Bayesian Neural Networks (BNNs) as fundamental mechanisms to generate diverse data in our prior.
Since our SCM and BNN priors only yield regression tasks, we show how to convert them to classification tasks in Section 3.5.
We describe additional refinements to our prior to reflect peculiarities of tabular data (correlated and categorical features, exponentially scaled data and missing values) better in Appendix C.2.
3. F UNDAMENTALLY P ROBABILISTIC M ODELS
Fitting a model typically requires finding suitable hyperparameters, e.g., the embedding size, number of layers and activation function for NNs.
Commonly, resource-intensive searches need to be employed to find suitable hyperparameters (Zoph and Le, 2017; Feurer and Hutter, 2019).
The result of these searches, though, is only a point estimate of the hyperparameter choice.
Ensembling over multiple architectures and hyperparameter settings can yield a rough approximation to a distribution over these hyperparameters and has been shown to improve performance (Zaidi et al., 2021; Wenzel et al., 2020).
This, however, scales linearly in cost with the number of choices considered.
In contrast, PFNs allow us to be fully Bayesian about our prior’s hyperparameters.
By defining a probability distribution over the space of hyperparameters in the prior, such as BNN architectures, the
PPD approximated by our TabPFN jointly integrates over this space and the respective model weights.
We extend this approach to a mixture not only over hyperparameters but distinct priors: we mix a
BNN and an SCM prior, each of which again entails a mixture of architectures and hyperparameters.
S IMPLICITY
We base our priors on a notion of simplicity, such as stated by Occam’s Razor or the Speed Prior (Schmidhuber, 2002).
When considering competing hypotheses, the simpler one is to be preferred.
x (1) x (2) z (1) z (2) z (3) z (4) y
(a)
[Figure]
BNN z (4) z (5) x (1) z (2) x (2) z (3) z (1) y
[Figure]
SCM z (3) z
(4) z (1) z (2) x (2) x (1) y z (3) z (4) x (1) x (2) z (2) x (3) z (1) y
[Figure]
Work in cognitive science has also uncovered this preference for simple explanations in human thinking (Wojtowicz and DeDeo, 2020).
Any notion of simplicity, however, depends on choosing a particular criterion that defines simplicity.
In the following, we introduce priors based on SCMs and
BNNs, in which simplicity materializes as graphs with few nodes and parameters.
3. SCM P RIOR
It has been demonstrated that causal knowledge can facilitate various ML tasks, including semisupervised learning, transfer learning and out-of-distribution generalization (Schölkopf et al., 2012;
Janzing, 2020; Rothenhäusler et al., 2018).
Tabular data often exhibits causal relationships between columns, and causal mechanisms have been shown to be a strong prior in human reasoning (Waldmann and Hagmayer, 2013; Wojtowicz and DeDeo, 2020).
Thus, we base our TabPFN prior on SCMs that model causal relationships (Pearl, 2009; Peters et al., 2017).
An SCM consists of a collection
Z :=
( { z 1 , . . . ,
z k } ) of structural assignments (called mechanisms)
[Equation]
where PA G ( i ) is the set of parents of z i (its direct causes) in an underlying DAG G (the causal graph), f i is a (potentially non-linear) deterministic function and ϵ
i is a noise variable.
Causal relationships in G are represented by directed edges pointing from causes to effects and each mechanism z i is assigned to a node in G , as visualized in Figure 2.
Predictions based on ideas from causal reasoning Previous works have applied causal reasoning to predict observations on unseen data by using causal inference, a method which seeks to identify causal relations between the components of a system by the use of interventions and observational data (Pearl, 2010; Pearl and Mackenzie, 2018; Lin et al., 2021).
The predicted causal representations are then used to make observational predictions on novel samples or to provide explainability.
Most existing work focuses on determining a single causal graph to use for downstream prediction, which can be problematic since most kinds of SCMs are non-identifiable without interventional data, and the number of compatible DAGs explodes due to the combinatorial nature of the space of DAGs.
In contrast, our TabPFN considers a broad family of SCMs and their respective weights but without any causal guarantees.
We skip any explicit graph representation in our inference step and approximate the PPD directly.
Thus, we do not perform causal inference but solve the downstream prediction task directly.
This implicit assumption of SCM-like processes generating our data can be explained in
Pearl's “ladder of causation”, an abstraction of inference categories, where each higher rung represents a more involved notion of inference (Pearl and Mackenzie, 2018).
At the lowest rung lies association, which includes most of ML.
At the second rung, we find predicting the effect of interventions, i.e., what happens when we influence features directly.
Our work can be considered as “rung 1.5”, similar to Kyono et al.
we do not perform causal reasoning, but make association-based predictions on observational data assuming SCMs model common datasets well.
In Figure 7, we experimentally show that TabPFN’s predictions indeed align with simple SCM hypotheses.
Defining a prior based on causal models
To create a PFN prior based on SCMs, we have to define a sampling procedure that creates supervised learning tasks (i.e., datasets).
Here, each dataset is based on one randomly-sampled SCM (including the DAG structure and deterministic functions f i ).
Given an SCM, we sample a set z X of nodes in the causal graph G , one for each feature in our
[Figure]
synthetic dataset, as well as one node z y from G .
These nodes are observed nodes: values of z X will be included in the set of features, while values from z y will act as targets.
For each such SCM and list of nodes z X and z y , n samples are generated by sampling all noise variables in the SCM n times (see Equation 3), propagating these through the graph and retrieving the values at the nodes z X and z y for all n samples.
Figure 2b depicts an SCM with observed feature- and target-nodes in grey.
The resulting features and targets are correlated through the generating DAG structure.
This leads to features conditionally dependent through forward and backward causation, i.e., targets might be a cause or an effect of features.
In Figure 3a, we show scatter plots of samples generated by two distinct SCMs, demonstrating the diversity in the space of datasets our prior can model.
In this work, we instantiate a large subfamily of DAGs and deterministic functions f i to build SCMs described in Appendix C.1.
Since efficient sampling is the only requirement we have, the instantiated subfamily is very general, including multiple activation functions and noise distributions.
3. BNN P RIOR
We also consider a BNN prior as introduced by Müller et al.
and mix it with the SCM prior described above by randomly sampling datasets during PFN training from either one or the other prior with equal probability.
To sample a dataset from the BNN prior, we first sample an NN architecture and its weights.
Then, for each data point in the to-be-generated dataset, we sample an input x , feed it through the BNN with sampled noise variables and use the output y as a target (see Figure 2a).We show an experimental ablation of the BNN prior based on Müller et al.
to our final mixture of both SCM-based and BNN-based priors prior in Appendix B.4.
M ULTI - CLASS P REDICTION
So far, the described priors return scalar labels.
In order to generate synthetic classification labels for imbalanced multi-class datasets, we need to transform our scalar labels ˆ y to discrete class labels y .
We do so by splitting the values of ˆ y into intervals that map to class labels:
i)
We sample the number of classes N c ∼ p ( N c ) , where p ( N c ) is a distribution over integers.
ii)
We sample N c − 1 class bounds B i randomly from the set of continuous targets ˆ y .
iii)
We map each scalar label ˆ y i to the index of the unique interval that contains it: y i ← (cid:80)
j [ B j < ˆ y i ] , where [ · ] is the indicator function.
For example, with N c = 3 classes the bounds B c = {− 0 . 1 , 0 . 5 } would define three intervals { ( −∞ , − 0 .
1] , ( − 0 .
5] , (0 . 5 , ∞ ) } .
Any ˆ y i would be mapped to the label
if it is smaller than − 0 .
1 , to 1 if lies in ( − 0 .
5] and to 2 otherwise.
Finally, we shuffle the labels of classes, i.e. we remove the ordering of class labels w.r.t.
the ranges.
T HE T AB PFN: A PFN F ITTED O N O UR N EW P RIOR F OR T ABULAR D ATA
Our TabPFN is a Prior-data Fitted Network (PFN, see Section 2) that is fitted on data sampled from our prior for tabular data (see Section 3).
We modify the original PFN architecture (Müller et al., 2022) in two ways.
i)
We make slight modifications to the attention masks, yielding shorter inference times.
ii) Additionally, we enable our model to work on datasets with different numbers of features by zero-padding.
These architectural enhancements are detailed in Appendix E.2.
In the prior-fitting phase, we train the TabPFN once on samples from the prior described in Section 3.
To be more precise, we trained a 12-layer Transformer for 18 000 batches of 512 synthetically generated datasets each, which required a total of 20 hours on one machine with 8 GPUs (Nvidia
RTX 2080 Ti).
This yielded a single network that is used for all our evaluations.
While this training step is moderately expensive, it is done offline, in advance, and only once for the TabPFN, as part of our algorithm development.
Full details about our TabPFN training are given in Appendix E.
During inference, the TabPFN approximates the PPD for our dataset prior, i.e., it approximates the marginal predictions across our spaces of SCMs and BNNs, including a bias towards simple and causal explanations for the data.
Looking back at the PPD equation (Equation 1), we can understand our TabPFN's predictions as follows:
The PPD integrates predictions over the space of hypotheses in our prior (which includes SCM graphs, BNN architectures, activation functions, etc.), where the weight of each hypothesis is determined by the product of the likelihood of observing the data given this hypothesis and the probability of this hypothesis in our prior.
In our experiments, we present predictions for a single forward pass of our TabPFN, as well as predictions that ensemble 32 forward passes of datasets augmented with shuffled feature columns and permuted class labels.
E XPERIMENTS
[Figure]
5. E VALUATION ON T OY P ROBLEMS
We first qualitatively compare our TabPFN to standard classifiers (without hyperparameter tuning) in Figure 4.
The top row shows the moons dataset with noise.
The TabPFN accurately models the decision boundary between samples; also, similar to Gaussian processes, its uncertainties are large for points far from observed samples.
The second row shows the circles dataset with noise: the TabPFN accurately models the circle's shape with high confidence anywhere outside the region where samples are mixed.
The third row shows two classes and features from the iris dataset, while the fourth rows shows two classes and features from the wine dataset (both included in scikit-learn ).
1s 5s 30s 5min 1h 0 .0 .0 .0 .0 .0 .0 .
Given Time Budget
M ea n R O
C
AU C 1s 5s 30s 5min 1h
Given Time Budget
M ea n R O
C
AU C
W i n s 1s 5s 30s 5min 1h
Given Time Budget
M ea n R O
C
AU C
R a nk
TabPFN
KNN
Reg.
Cocktails
LogReg
Catboost
SAINT
LightGBM
XGBoost
Autogluon
Auto-sklearn [Figure]
E VALUATION ON T ABULAR ML T ASKS
Now, we turn to an empirical analysis of our method for real-world classification tasks.
We compare our method against state-of-the-art ML and AutoML methods for tabular classification.
Datasets
As test datasets, we used all datasets from the curated open-source OpenML-CC benchmark suite (Bischl et al., 2021) that contain up to 2 000 samples, 100 features and 10 classes.
The resulting set comprises 30 datasets.
We focus on small datasets because (1) small datasets are often encountered in real-world applications (Dua and Graff, 2017), (2) existing DL methods are most limited in this domain (Grinsztajn et al., 2022) and (3) the TabPFN would be significantly more expensive to train and evaluate for larger datasets, detailed in Appendix A.
Baselines We compare against five standard ML methods and two state-of-the-art AutoML systems for tabular data.
As ML models we considered two simple and fast baselines, K-nearest-neighbors (KNN) and Logistic Regression (LogReg) .
Additionally, we considered three popular tree-based boosting methods, XGBoost (Chen et al., 2020), LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018).
For each ML model, we used 5 -fold cross-validation to evaluate randomly drawn hyperparameter configurations until a given budget was exhausted or 10 000 configurations were evaluated (for the search spaces, see Appendix F.2).
We then chose the best-performing hyperparameter configuration (maximum ROC AUC OVO) and refit on the whole training set.
Where necessary, we imputed missing values with the mean, one-hot encoded categorical inputs, normalized features and passed categorical feature indicators.
As more complex but powerful baselines, we chose two state-of-the-art AutoML systems: AutoGluon (Erickson et al., 2020), which combines ML models including neural networks and tree-based models into a stacked ensemble, and Auto-sklearn 2.0 (Feurer et al., 2015; 2021) which uses Bayesian Optimization and combines the evaluated models into a weighted ensemble.
We note that previous works have found that DL baselines do not outperform or match the performance of GBDT or AutoML methods for small to medium-sized tabular data (< 10,000 samples; while matching GBDT performance on larger datasets) (Borisov et al., 2021;
Grinsztajn et al., 2022; Shwartz-Ziv and Armon, 2022).
Furthermore, DL methods such as TabNet, SAINT, Regularization Cocktails, Non-parametric Transformers (Arik and Pfister, 2021; Somepalli et al., 2021; Kadra et al., 2021; Kossen et al., 2021) are evaluated on much larger
[Table]
TabPFN n.e. is a faster version of our method, without any ensembling, while TabPFN + AutoGluon is an ensemble of TabPFN and AutoGluon.
Separate inference times can be found in Table 2.
datasets and often use custom parameter tuning and preprocessing.
We still evaluate two prominent
DL methods: Regularization Cocktails 3 and SAINT (Kadra et al., 2021; Somepalli et al., 2021).
Evaluation Protocol For each dataset and method, we evaluated 5 repetitions, each with a different random seed and a train- and test split (50% train and 50% evaluation samples; all methods used the same split given a seed).
To aggregate results across datasets, we report the ROC AUC (one-vs-one (OVO) for multi-class classification) average, ranks and wins including the 95% confidence interval and compare to the performance of the baselines with a budget of { 30 , 60 , 300 , 900 , 3600 } seconds.
Our TabPFN uses 32 data permutations for ensembling as described above; we also evaluate TabPFN without permutations, which we label “TabPFN n.e. ”
in Table 1.
Results We present the achieved
AUC ROC as a function of budget in Figure 5, demonstrating that TabPFN is dramatically faster than all the other methods: it makes predictions within less than a second on one GPU that tie with the performance of the best competitors (the AutoML systems) after training one hour, and that dominate the performance of tuned GBDT methods.
Unsurprisingly, the simple baselines ( LogReg , KNN ) already yield results with a small budget but perform worst overall.
GBDTs ( XGBoost , CatBoost ) perform better but are clearly outperformed by the state-of-the-art AutoML systems ( Auto-sklearn 2.0 , Autogluon ).
We could see that while TabPFN's performance is different depending on the kind of dataset though.
It is especially strong for purely numerical datasets, while being not as strong when categorical features are present.
Appendix B.1 presents a more detailed overview of the results.
TabPFN is much faster than comparably performing methods.
In the following we compare the times for training and prediction (and tuning if applicable) together.
TabPFN without ensembling requires 0.0187s on a GPU or 0.87s on a CPU on average to predict for one dataset.
It performs comparably to the strongest baselines at one minute, yielding a 70 × speedup on CPU and a 3 200 × speedup using a GPU.
Given more budget ( ≥ 5 minutes), the strongest baseline achieves competitive performance with the original TabPFN (including ensembling), which uses a time budget of 0.
seconds on a GPU or 24.5s on a CPU.
If we compare TabPFN against the 5 minutes required for the baseline, it yields a 12 × speedup on a CPU and a 710 × speedup using a GPU.
These times assume that our trained model is in (GPU) memory already, which otherwise required 0.2s for us.
We ignore computational development costs of each method, see Appendix F.5 for why.
Results on the OpenML-AutoML Benchmark After finishing our above analysis and releasing the pre-trained final TabPFN, we also evaluated it on the small ( ≤ 1 000 training samples, 100 features,
10 classes) datasets of the OpenML-AutoML Benchmark, for which externally-validated performance numbers are available.
We use the setup provided by the OpenML-AutoML Benchmark, i.e.: metrics, official evaluation scripts and pre-released evaluations for an even wider range of AutoML baselines.
While using only an average of 4.4 seconds per dataset on a single CPU, compared to 60 minutes for the baselines, TabPFN outperformed all baselines in terms of mean cross-entropy, accuracy and the OpenML Metric (which evaluates binary classification datasets using ROC AUC and multiclass datasets using Cross Entropy).
We provide detailed results in Appendix B.2.
In-depth analysis of TabPFN predictions
We evaluated our model predictions in a plethora of ways to provide extra insights, such as the inductive biases of our method, see Appendix B.3.
We confirm that TabPFN learns to make predictions biased towards simple causal explanations, as detailed in B.3.1, while GBDT methods do not share this inductive bias.
We also evaluate our method's invariance to feature rotations and robustness to uninformative features in Appendix B.3.
and B.3.3.
In Figure 6, we observe that TabPFN is especially strong compared to baselines when datasets do not contain categorical features or missing values.
Ensembling We observe that TabPFN performs best on different datasets than our baselines, i.e., the per-dataset correlation of ROC AUC scores between TabPFN and the strong baselines is lower than between the strong baselines, visualized in Figure 10 in our Appendix.
This is likely due to the novel inductive biases of our approach, which lead to distinct predictions.
Ensembling predictions is more effective when the considered methods make fewer correlated errors (Breimann, 2001), which encourages the use of more diverse strategies, making TabPFN an ideal candidate for ensembling with baseline methods (Wu et al., 2021).
Also, TabPFN is evaluated so quickly, that predictions are almost free, compared to the long runtimes of the baselines.
To demonstrate this, we include an entry "TabPFN + AutoGluon" in Table 1, generated by averaging the predictions of TabPFN and
AutoGluon ; this strongly outperforms all other methods.
Model Generalization
The Transformer-based PFN architecture accepts datasets of any length as input.
However, when training our model in the synthetic prior-fitting phase, we limited synthetic datasets to a maximum size of 1 024 .
We made this choice because generating datasets and performing gradient updates is increasingly expensive for larger datasets.
Nevertheless, we wondered:
Would TabPFN generalize to larger training set sizes that were never seen during training?
To test this, we used a collection of 18 datasets from the OpenML-AutoML Benchmark (see Table 11) from which we selected 10 000 samples each.
We evaluated TabPFN on up to 5 000 training samples, usingrandom data splits and 5 000 test samples for each size.
Surprisingly, our models generalize to sample sizes that were never seen during training, as shown in Figure 9.
That is, average ROC AUC improves when more training data is provided.
C ONCLUSIONS & F UTURE W ORK
We have shown how a single Transformer, the TabPFN, can be trained to do the work of a full
AutoML framework for tabular data and can yield predictions in 0.4 seconds that are competitive with the performance that the best available AutoML frameworks achieve in 5 to 60 minutes.
This result could disrupt the field of data science since it allows real-time fitting and prediction.
It also slashes the computational expense of AutoML, enabling affordable, green, AutoML.
The TabPFN still has important limitations: the underlying Transformer architecture only scales to small datasets as detailed in Appendix A; our evaluations were done on classification datasets with only up to 1 000 training samples, 100 features and 10 classes, which motivates work on (1) scaling up to large datasets.
Our in-depth analysis of the inductive biases of TabPFN (see Appendix B.3), points towards extensions for (2) improved handling of categorical features.
Also, our work motivates a multitude of exciting follow-ups regarding the (3) handling of missing values, (4) integration of
TabPFN into existing AutoML frameworks; (5) ensembling to continue making improvements given more time; (6) dataset-dependent choices of the prior; (7) generalizations to non-tabular data and (8) regression tasks, as well as studies of the TabPFN regarding the dimensions of trustworthy AI (e.g., (9) out-of-distribution robustness; (10) algorithmic fairness, (11) robustness to adversarial examples, (12) explainability, and (13) auditability).
The almost instant state-of-the-art predictions of TabPFN are also likely to give rise to (14) novel exploratory data analysis methods, (15) novel feature engineering methods and (16) novel active learning methods.
Finally, our advances in causal reasoning warrant follow-ups on (17) approximating the effects of interventions and counterfactuals considering a distribution of SCMs.
E THICS S TATEMENTS
In terms of broader societal impact of this work, we do not see any foreseeable strongly negative impacts.
However, this paper could positively impact the carbon footprint and accessibility of learning algorithms.
The computations required for machine learning research have been doubling every few months, resulting in a large carbon footprint (Schwartz et al., 2020).
Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers to apply these methods.
The decreased computational time shown by TabPFN translates to reductions in CO2 emissions and cost, making it available to an audience that does not have access to larger scale computing.
As the TabPFN provides a highly portable and convenient way of building new classifiers that work in real-time, it is likely to increase the pervasiveness of machine learning even further.
While this can have many positive effects on society, such as better personalized healthcare, increased customer satisfaction, efficiency of processes, etc, it will also be crucial to study and improve the TabPFN under the lens of the many dimensions of trustworthy AI other than computational sustainability, such as algorithmic fairness, robustness to adversarial examples, explainability, and auditability.
We hope that its foundation in causal models and simplicity will allow possible avenues for work along these lines.
R EPRODUCIBILITY
Code release In an effort to ensure reproducibility, we release code alongside our pretrained TabPFN and notebooks to reproduce our experiments at https://github.com/ tabpfn-anonym/TabPFNAnonym .
Application to public benchmarks In our work, we evaluate TabPFN to publicly available benchmarks: the OpenML-CC18 Benchmark and the OpenML-AutoML Benchmark.
This ensures, that dataset choices are not cherry-picked to our method.
Also, for the OpenML-AutoML Benchmark, we use official baseline results and evaluate our method using the evaluation scripts published for this benchmark.
Availability of datasets All datasets used in our experiments are freely available at OpenML.
org (Vanschoren et al., 2014), with downloading procedures included in the submission.
Further details on the datasets used can be found in Section F.3.
Online resources We created a Colab notebook, that lets you interact with our scikit-learn interface at https://colab.research.google.com/drive/1J0l1AtMV_
H1KQ7IRbgJje5hMhKHczH7-?usp=sharing .
We created another Colab, where our evaluation and plots on 179 test and validation datasets can be reproduced easily.
https://colab.research.google.com/drive/ 1yUGaAf3D7RSyO5Jc4PYXSVbtaUAozh6S
We also created two demos.
One to experiment with the TabPFNs predictions ( https: //huggingface.co/spaces/TabPFN/TabPFNPrediction ) and one to check crossvalidation ROC AUC scores on new datasets ( https://huggingface.co/spaces/TabPFN/
TabPFNEvaluation ).
Both of them run on a weak CPU, thus it can require a little bit of time.
Details of training procedures for TabPFN and baselines Details shared in the training procedure of all our Transformer models can be found in Appendix F.
An overview of the hyperparameters used for running the TabPFN and our baselines can be found in Tables 5 and 6, respectively.
A L IMITATIONS
The runtime and memory usage of the Transformer-based PFN architecture used in this work scales quadratically with the number of inputs, i.e. training samples passed.
Thus, inference on larger sequences (> 100 000 ) is hard on current consumer GPUs.
A growing number of methods seek to tackle this issue and report similar performances while scaling linearly with the number of inputs (Zaheer et al., 2020; Beltagy et al., 2020).
These methods can be integrated into the PFN architecture and thus into the TabPFN.
Furthermore, in our experiments we limit the number of features toand the number of classes to 10 as described in Section 5.
While this choice is flexible, the precise
TabPFN that we fitted cannot work with datasets that go beyond these limits.
While baseline models (except Gaussian Processes) fit and inference in separate steps, TabPFN performs both at the same time.
Thus when pretrained baseline models are used, inference is fast (see Table 2).
B A DDITIONAL R ESULTS B. D ETAILED T ABULAR R ESULTS
[Figure]
Autosklearn
Autogluon
Catboost
[Figure]
Reg.
[Figure]
In Figure 6, we explore how the kind of dataset evaluated affects the performance of TabPFN, compared to our baselines.
We find that TabPFN performs much better when no categorical features are present.
We also find that TabPFN performs better, when no missing values are present in the data.
This warrants an extension of our prior in future work, to make it more customized towards categorical and missing data.
Our method seems to work comparably well for binary and multi-class problems.
In addition to the results in the main paper in Section 5.2, we report a wide range of performance values and per dataset results in Table 2.
[Table]
B. R ESULTS ON THE O PEN ML-A UTO ML B ENCHMARK
We evaluate TabPFN using the official benchmarking scripts, 6 datasets, splits and baselines results of the OpenML-AutoML Benchmark.
The full list of datasets can be found in Table 10.
We note that these datasets are not disjoint from our evaluation datasets, in fact 4 evaluated datasets from the
OpenML-AutoML Benchmark were included in our evaluation datasets from the OpenML-CC
Benchmark as well, while one dataset (“Australian”) was included in our list of 150 meta-validation datasets.
The evaluation on another set of benchmarking scripts with previously published train-test splits and baseline results, however, help to confirm that: (1) Our baselines are well tuned and not outperformed by another method in the extensive OpenML-AutoML Benchmark; (2) the runtimes of
TabPFN are reproducible in controlled environment provided by the OpenML-AutoML Benchmark; and (3) TabPFN is not overfit to datasplits or our evaluation metric.
While we used a 50-50 train-test split with 5 iterations for our experiments in Table 3, the OpenML-AutoML Benchmark uses a 10-fold cross-validation, which results in a 90-10 splits with 10 iterations.
Thus in the OpenML-AutoML
Benchmark, all methods use more training samples as in our experiments on the OpenML-CC
Benchmark, which leads to slightly stronger results.
[Table]
B. I N - DEPTH ANALYSIS OF MODEL BIASES
Previous work by Grinsztajn et al.
(2022) empirically investigates the inductive biases of tree-based and deep-learning models.
They identify three challenges in developing tabular-specific models.
Models must be (1) able to fit irregular functions; (2) robust to uninformative features; and (3) preserve the orientation of the data.
We perform and extend this analysis in the following section.
We consider three model types in our analyses: TabPFN, GBDTs (LightGBM) and NNs (Standard Sklearn
Multi Layer Perceptron with a hidden deimensionality of 100).
We do not seek to make absolute comparisons between these model types, which would warrant tuning of our baselines, but only seek to investigate their qualitative behavior in the following experiments.
B.3. F ITTING IRREGULAR PATTERNS
Complexity
N u m b e r
o f s a m p l e s
Log.
Regr.
Complexity
LightGBM
Complexity
Tabular PFN 0.
While GBDT methods learn non-smooth and irregular patterns in the targets, MLPs learn smooth, low-frequency functions, as can be seen in Figure 4.
The work by (Grinsztajn et al., 2022) suggests that GBDT methods perform well since they are able to learn non-smooth patterns.
Prior works on regularization and an intuitive exploration of the target decision boundary of GBDT methods in Figure 4, suggests , however, that smooth target functions are desirable.
TabPFN learns target functions that are rather smooth, as suggested in Figure 4.
This is due to our model´s prior, which prefers simple SCMs as explanations and thus less irregular decision planes.
We note, however, that when many training samples are provided, TabPFN fits more complex functions.
The tradeoff between complexity and number of training samples is explored in Figure 7.
Here, we evaluate the training set uncertainty (cross-entropy loss) on synthetic data generated from random
SCMs.
The number of training samples and the complexity of the generated data (number of hidden units in the data generating graph) is varied.
B.3. R OBUSTNESS TO UNINFORMATIVE FEATURES
[Figure] [Figure]
Method
R O
C
AU
C
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
lgbmlptabpfn
Percentage
[Figure]
features
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
lgb mlp tabpfn
[Figure]
Tabular datasets contain a large fraction of uninformative features (Grinsztajn et al., 2022).
To evaluate robustness to uninformative features, we add an increasingly large fraction of uninformative features to our data and show results in Figure 8.
Uninformative features are generated by copying existing features and shuffling their values randomly between samples.
We find that TabPFN and MLPs are less robust to uninformative features than LightGBM.
TabPFN could be adapted by including more uninformative features in the used prior.
In a second experiment we drop an increasingly large fraction of features.
We drop these features according to feature importance (ranked by a Random
Forest), first removing least informative features.
We show results in Figure 8 and observe that the classification accuracy of a TabPFN and GBDT is not much affected by removing up to 30% of the features, but constantly diminishes.
The MLP, which is less not robust to uninformative features, initially performs even better when features are removed.
We use our testing tasks from the OpenML CC-18 Benchmark for all evaluations.
To simplify analyses, we drop multiclass datasets and datasets that contain more than 50 features (As adding 100% more features to a dataset with more than 50 features yields more than 100 features, which
TabPFN cannot handle).
B.3. I NVARIANCE TO FEATURE ROTATION
Each feature of a tabular dataset typically carries meaning individually, as expressed by column names, such as age or sex.
A learning algorithm is rotationally invariant in the sense of Ng (2004), if it is left unchanged when a rotation (unitary) matrix is applied to the features of both the training and testing set, i.e. when features are mixed.
To remove uninformative features under a feature rotation, an algorithm first has to restore the original orientation of the features, and then select informative ones.
A rotationally invariant algorithm discards the data orientation and thus has to restore the original orientation internally.
Thus, Ng (2004) shows that any rotationally invariant
[Figure]
learning algorithm has a worst-case sample complexity that grows at least linearly in the number of irrelevant features.
Figure 8 shows the change in test ROC AUC when randomly rotating our datasets, and confirms that only MLPs are rotationally invariant.
GBDT methods are highly sensitive to rotations, while TabPFN is less sensitive, but still performs better when no rotations are applied.
The theoretical results by Ng (2004) and empirical results by Grinsztajn et al.
(2022) imply, that
TabPFN's diminishing performance when uninformative features are added is linked to it's relative rotation invariance.
Adjusting the prior to include more uninformative features could address these results.
We use our testing tasks from the OpenML CC-18 Benchmark for all evaluations.
To simplify analyses, we drop multiclass datasets and datasets that contain categorical data.
Rotating categorical datasets is problematic, as some GBDT classifiers treat categorical variables distinctly (e.g. generating embeddings per category).
B. A BLATION ON THE SELECTION OF PRIOR MODELS
We perform ablation experiments for a prior based solely on BNNs, SCMs and a mix of both using a hyperparameter to control the sampling likelihood for either.
The PFNs for each prior are fitted using less compute than in our final experiments, thus their scores generally are slightly worse.
The BNN prior, similar to the one used in Müller et al.
(2022), provides diminished performance compared to the priors based on causal models.
Additionally, we can see that mixing BNN and SCM prior does not seem to make a big difference on our test set compared to a pure SCM prior.
BNN
[Table]
B. E XTENDED ANALYSIS ON A LARGER BENCHMARK OF DATASETS
We provide an extended benchmark where we:
[Figure]
see
i) Include an analysis on the additional 149 validation datasets 7 (see 8).
ii) Include default configurations of our baselines.
iii)
Run one step of the default configuration of our baselines initially and then tuning them.
iv)
Added SVM, Gradient Boosting and Random Forests as a Baseline.
and report results in Figure 13 and Figure 14.
The benchmark was done across 5 splits and followed the same experimental setup described in F.
C D ETAILS OF THE T AB PFN P RIOR C. SCM P RIOR
The Sampling Algorithm We instantiate a subfamily of DAGs that can be efficiently sampled from by starting with a MLP architecture and dropping weights from it.
That is, to sample a dataset with k features and n samples from our prior we perform the following steps for each dataset:
(1) We sample the number of MLP layers l ∼ p ( l ) and nodes h ∼ p ( h ) and sample a graph
G
( Z , E ) structured like an l -layered MLP with hidden size h .
(2) We sample weights for each Edge E ij as W i,j ∼ p w ( · ) .
(3) We drop a random set of edges e ∈ E to yield a random DAG.
(4) We sample a set of k feature nodes N x and a label node N y from the nodes Z .
We sample the noise distributions p ( ϵ ) ∼ p ( p ( ϵ )) from a meta-distribution.
This yields an SCM, with all f i ’s instantiated as random affine mappings followed by an activation.
Each z i corresponds to a sparsely connected neuron in the MLP.
With the above parameters fixed, we perform the following steps for each member of the dataset:
(1) We sample noise variables
ϵ i from their specific distributions.
(2) We compute the value of all z ∈ Z with z i = a (( (cid:80) j ∈ PA G ( i ) E ij z j )
+ ϵ i ) .
(3) We retrieve the values at the feature nodes N x
and the output node N y and return them.
We sample one activation function a per dataset from { Tanh, LeakyReLU, ELU, Identity } (Nair and Hinton, 2010).
The sampling scheme for the number of layers p ( l ) and nodes p ( h ) is designed to follow a discretized noisy log-normal distribution, p ( ϵ ) is a noisy log-normal distribution and the dropout rate follows a beta distribution.
The full information can be found in Table 5.
C. T ABULAR D ATA R EFINEMENTS
Tabular datasets comprise a range of peculiarities, e.g. feature types can be numerical, ordinal, or categorical and feature values can be missing, leading to sparse features.
We seek to reflect these peculiarities in the design of our prior as described in the following sections.
C.2. P REPROCESSING
During meta-training, input data is normalized to zero mean and unit variance, and we apply the same step when evaluating on real data.
Since tabular data frequently contains exponentially scaled data, which might not be present during meta-training, we apply power scaling during inference (Yeo and Johnson, 2000).
Thus, during inference on real tabular datasets the features more closely match those seen during meta-training.
We use only training samples for calculating z-statistics, power transforms and all other preprocessing.
We take this preprocessing time into account when reporting the inference time of our method.
C.2. C ORRELATED F EATURES
Feature correlation in tabular data varies between datasets and ranges from independent to highly correlated.
This poses problems to classical deep learning methods (Borisov et al., 2021).
When considering a large space of SCMs, correlated features of varying degrees naturally arise in our priors.
Furthermore, in real-world tabular data, the ordering of features is often unstructured, however adjacent features are often more highly correlated than others.
We use “Blockwise feature sampling” to reflect the correlation structure between ordered features.
Our generation method of SCMs naturally provides a way to do this.
The first step in generating our SCMs is generating a unidirectional layered network structure in which nodes in one layer can only receive inputs from the preceding layer.
Thus, features in the same layer tend to be more highly correlated.
We use this by sampling adjacent nodes in the layered network structure in blocks and using these ordered blocks in our set of features.
In
Figure 11, we visualize the correlations of such a generated dataset correlation
C.2. G ENERATING IRREGULAR FUNCTIONS
In real-world data, some features are consistently more important than others.
While a random network weight initialization leads to slightly different feature importances, the average effect of input features regresses to the mean when the hidden dimensionality increases.
We amplify differences by sampling a weight parameter for each input feature and multiplying all outgoing weights by this factor.
In the prior, we randomly sparsify connections of the graph.
Thus hidden variables and the output node are influenced by fewer parameters, yielding more irregular patterns, as a larger number of parameters once again regresses to the mean.
We also extend sparsification to blocks of variables, leading to some groups of variables interacting more strongly.
We also extend the way noise variables are sampled.
Instead of sampling Gaussian noise at each node from the same distribution, we first sample separate noise means and standard deviations for each node and then sample from this distribution.
Also, we generate non-uniformly distributed input data x, as observed in real-world data: We sample the input variables x (which are propagated through our network), from a mix of distributions, namely the Gaussian, Zipfian and Multivariate Distribution.
C.2. N AN H ANDLING
We do not have special nan handling built into our model.
We replace nan values with zero at test time.
C.2. C ATEGORICAL F EATURES
Tabular data often includes not only numeric features but also discrete categorical ones.
While categorical features should technically not be ordered, in practice, they sometimes are, i.e., the categories represent binned degrees of some underlying variable.
We introduce categorical features by picking a random fraction p cat (a hyperparameter) of categorical features per dataset.
Analogous to transforming numeric class labels to discrete multiclass labels, we convert dense features to discrete ones.
Also analogous to multiclass labels, we pick a shuffling fraction of categorical features p scat where we reshuffle categories.
For details, see Section 3.5.
During prior-fitting, we use a probability for categorical features of 20% .
C.2. P RIOR WORK ON PFN S FOR T ABULAR D ATA
Prior work by Müller et al.
(2022) has demonstrated tabular data classification using PFNs, but was limited to 30 training samples, balanced binary-classification and 60 features.
Here we summarize the most important changes, to this prior work:
i)
The PFNs for tabular data described in Müller et al.
(2022) can only handle balanced binary datasets, in Section 3.5 we show how to extend the prior s.t.
we can handle unbalanced data.
ii)
We looked into pre-processing techniques for the TabPFN, which was not done at all before.
Finally, we ensemble over different pre-processing pipelines, which include (power transforms and outlier removal) and feature/class rotations.
iii)
We changed the transformer architecture s.t.
it is faster.
We shrank attention matrix sizes from ( n + m ) 2 to n 2 + n ∗ m , for n training points and m inference points.
iv)
We introduce a novel SCM prior.
We compare the SCM, the improved BNN prior (whcih is more heavily based on the BNN setup of C.2.6) and SCM + BNN in Table 4.
It pushes performance by 2% in this smaller scale setup, which is a bigger difference than between the final TabPFN and all baselines besides KNN and SAINT.
D D ETAILS OF THE P RIOR -D ATA F ITTED N ETWORK A LGORITHM
Algorithm 1 describes the training method proposed by Müller et al.
(2022) for PFNs.
Algorithm 1: Meta
Training of a PFN (Müller et al., 2022)
Input datasets
, from which samples can be drawn and the number of samples K to draw
Output : will approximate the PPD
Initialize the neural network q θ ; for j ← 1 to K do
Sample D ∪ { ( x i , y i ) } mi =1 ∼ p ( D ) ;
Compute stochastic loss approximation ¯ ℓ θ = (cid:80) mi =1 ( − log q θ ( y i | x i , D )) ;
Update parameters θ with stochastic gradient descent on ∇ θ ¯ ℓ θ ; end
E S ETUP OF OUR METHOD E. T RANSFORMER H YPERPARAMETERS
We considered only PFN Transformers with 12 layers, embeddings size 512 , hidden size 1024 in feed-forward layers, and 4 -head attention.
We used the Adam optimizer (Kingma and Ba, 2015) with linear-warmup and cosine annealing (Loshchilov and Hutter, 2017).
For each training we tested a set of 3 learning rates, { . 001 , .
0001 } , and used the one with the lowest final training loss.
The resulting model contains 25.82 M parameters.
E. PFN A RCHITECTURE A DAPTATIONS
Attention Adaption
The original PFN architecture (Müller et al., 2022) uses a single multi-head selfattention module (Vaswani et al., 2017) to compute the attention between all the training examples, as well as, the attention from validation examples to training examples.
We replaced this, with two modules that share weights, one which computes self-attention among the training examples and the other that only compute cross-attention from validation examples to training examples.
Conceptually, this is equivalent to the original architeture, except that we’re using a slightly different self-attention mask than the original architecture, which allowed all examples to attend to itself (the diagonal is 1), as in this example:
[Equation]
For validation examples, we remove the attention to themselves.
In terms of the example above:
[Equation] [Equation]
Information about the state of the current position does still flow through the residual branch, though.
Flexible Encoder Datasets have unequal numbers of input dimensions (features), while PFNs use an encoder layer that accepts fixed dimensional inputs.
Here we explain how datasets with different numbers of dimensions can be modelled with a single PFN: We draw the number of dimensions of a dataset during training uniformly at random up to 100 .
Our encoder changes to accomodate this training and inference with different numbers of features by zero-padding datasets where the number of features k is smaller than the maximum number of features K and scaling these features by Kk , s.t.
the magnitude stays the same.
E. T AB PFN T RAINING
We trained our final model for 18 000 steps with a batch size of 512 datasets.
That is our TabPFN is trained on 9 216 000 synthetically generated datasets.
This training takes 20 hours on 8 GPUs (Nvidia
RTX 2080 Ti).
Each dataset had a fixed size of 1024 and we split it into training and validation uniformly at random.
We generally saw that learning curves tended to flatten after around 10 million datasets and were generally very noisy.
Likely, this is because our prior generates a wide variety of different datasets.
E. P RIOR H YPERPARAMETERS
The hyperparameters of our prior were chosen based on simplicity and our observations on the validation datasets (such as their class distributions or feature correlation strengths).
Also, during algorithm development, we evaluated our models on this set of datasets to decide if our developed methods were correct and working.
Since our prior hyperparameters specify distributions and not definite values, they can be chosen over a wide range and resemble the intervals chosen for a random hyperparameter search.
The prior distributions we used are given in Table 5.
[Table] [Table]
[Table] [Table]
[Table]
F D ETAILS FOR T ABULAR E XPERIMENTS
Here we provide additional details for the experiments conducted in Section 5 in the main paper.
F. H ARDWARE S ETUP
All evaluations, including the baselines, ran on a compute cluster equipped with Intel(R)
Xeon(R)
Gold 6242 CPU @ 2.80GHz using 1 CPU with up to 6 GB RAM.
For evaluation using our TabPFN, we additionally use an RTX 2080 Ti.
F. B ASELINES
We provide the search space used to tune our baselines in Table 6.
For CatBoost and XGBoost , we used the same ranges as Shwartz-Ziv and Armon (2022) with the following exception: For CatBoost we removed the hyperparameter max_size since we could not find it in the official documentation.
To be maximally fair to XGBoost, we also tried the search space of quadruple Kaggle grandmaster
Bojan Tunguz (Tunguz, 2022), which we adapted slightly by using softmax instead of logistic, as we are in the multi-class setting.
XGBoost with this search space performed worse for all considered time budgets than the search space by Shwartz-Ziv and Armon (2022).
The search spaces for the
KNN, GP and Logistic Regression baselines were designed from scratch and we used the respective implementation from scikit-learn (Pedregosa et al., 2011).
For CatBoost and AutoSklearn , we pass the position of categorical features to the classifier ( AutoGluon automatically detects categorical feature columns).
We normalize inputs for Logistic Regression, GP and KNN to the range [0, 1] using MinMax Scaling.
F. U SED D ATASETS
To construct and evaluate our method, we used the following four sets of datasets.
First, our meta-test set (see Table 7) comprises all datasets in the OpenML-CC18 benchmark suite (Bischl et al., 2021)(available at OpenML.org ) with at most 2 000 samples, 100 features and 10 classes, which leaves us with 30 datasets that represent small, tabular datasets.
[Table]
b a l a n c es c a l e m f e a t - f o u r i e r b r e a s t - w m f e a t - k a r hun e n m f e a t - m o r p
h
o l
o g
i c a l m f e a t - z e r n i k
e c m c c r e d
i t - a pp r o v a l c r e d i t - g d i a b e t e s t
i c - t a c - t o e v e h
i c
l e e
u c a l y p t u s a n a l c a t d a t a _ a u t h
o r s h
i p a n a l c a t d a t a _ d m f t p c
4 p c 3 k c 2 p cb a n k n o t e - a u t h e n t
i c a t i o n b l oo d - t r a n s f u s i o n - s e r v
i c
e - c e n t e r il pd q s a r - b i o d e g w db c c y li n d e rb a n d s d r e ss e ss a l e s
M i c e P r o t e i n c a r s t ee l - p l a t e s - f a u l t c li m a t em o d e l - s i m u l a t
i o n - c r a s h e s 0.
[Figure]
baseline for each dataset.
Ensemble weights are averaged across 5 splits for each dataset
OpenML-CC18 Benchmark hour training.
Second, our meta-validation set (see Tables 8 and 9) comprises 150 datasets from OpenML.org (Vanschoren et al., 2014).
For this, we considered all datasets on OpenML.org and applied the following filtering procedure: We dropped all datasets that are in the meta-test set and all datasets with more than 2 000 samples, 100 features or 10 classes.
We also manually checked for overlaps and removed datasets where the number of features, classes and samples was identical to a dataset in the meta-test set.
Furthermore, we manually dropped FOREX (since it is a time series dataset) and artificiallycreated datasets, such as the Univ and Friedman datasets.
The remaining meta-validation set then contains 150 datasets.
This meta-validation set was used to guide the development of our prior hyperparameters as described in Appendix E.4.
Third, a subset of 5 datasets from the OpenML-AutoML Benchmark, comprises all datasets from the
OpenML-AutoML Benchmark with at most 1 111 samples, 100 features and 10 classes.
This is given in Table 10.
Due to the 10-fold cross-validation in the OpenML-AutoML Benchmark this is identical to the setup for our meta-test and meta-validation datasets above, where we used up to 2 000 samples split 50-50 into training and test.
Fourth, our meta-generalization set, described in Appendix F.4, comprises 18 larger datasets from the
OpenML AutoML Benchmark.
F. M ODEL G ENERALIZATION
For testing the TabPFN performance on longer sequences, as described in Section 9, we used a set of 18 datasets from the OpenML AutoML Benchmark that contain at least 10 000 samples.
The list of datasets used can be found in Table 11.
For this evaluation, datasets with more than 100 features are limited to the first 100 features.
When more than 10 classes are contained in the datasets, samples with any but the first 10 classes are discarded.
F. D ETAILS ON T IME C OMPARISONS
Time comparisons refer to combined fitting, tuning and prediction; See Table 2 for the times split into tuning/fitting and prediction. The time taken for each baseline and TabPFN does not include the one-time cost of development of each method. Thus for AutoML baselines meta-learning cost was not included (e.g. Auto-Sklearn pipelines meta-learning, which involved running hyper-parameter search for 24 hours on 140 datasets (= 3360 CPU hours) (Feurer et al., 2021)). For GBDT methods the manual time that went into defining suitable hyperparameter spaces and the manual crafting of algorithms that perform well on tabular datasets is hard to measure and was not included.
For TabPFN the prior-fitting phase (which is part of our algorithm development: i
[Figure]
e. developing ideas, writing code, trying ideas) is not included. This is fair to all methods, as these costs are not on the user side and are amortized over time
[Figure]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
that TabPFN is much stronger on purely
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
disclosure
[Table]
774 bodyfat      778 cleveland      786 triazines      788 disclosure_x_tampered      795 cpu      796 cholesterol      798 chscase_funds      801 pbcseq      802 pbc      810 rmftsa_ctoarrivals      811 chscase_vine       814 chatfield_       820 boston_corrected      825 sensory      826 disclosure_x_noise      827 autoMpg      831 kdd_el_nino-small      839 autoHorse      840 stock      841 breastTumor      844 analcatdata_gsssexsurvey      852 boston      853 fishcatch      854 vinnie      860 mu
no       886 chscase_geyser
[Figure]
[Table] [Table]
[Table]