T AB PFN: A T RANSFORMER T HAT S OLVES S MALL T ABULAR C LASSIFICATION P ROBLEMS IN A S ECOND
A BSTRACT
We present TabPFN, a trained Transformer that can do supervised classification for small tabular datasets in less than a second , needs no hyperparameter tuning and is competitive with state-of-the-art classification methods.
TabPFN is fully entailed in the weights of our network, which accepts training and test samples as a set-valued input and yields predictions for the entire test set in a single forward pass.
TabPFN is a Prior-Data Fitted Network (PFN) and is trained offline once, to approximate Bayesian inference on synthetic datasets drawn from our prior.
This prior incorporates ideas from causal reasoning:
It entails a large space of structural causal models with a preference for simple structures.
On the 18 datasets in the OpenML-CC18 suite that contain up to 1 000 training data points, up to 100 purely numerical features without missing values, and up to 10 classes, we show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to 230 × speedup.
This increases to a 5 700 × speedup when using a GPU.
We also validate these results on an additional 67 small numerical datasets from OpenML.
We provide all our code, the trained TabPFN, an interactive browser demo and a Colab notebook at https://github.com/automl/TabPFN .
I NTRODUCTION
Tabular data has long been overlooked by deep learning research, despite being the most common data type in real-world machine learning (ML) applications (Chui et al., 2018).
While deep learning methods excel on many ML applications, tabular data classification problems are still dominated by
Gradient-Boosted Decision Trees (GBDT; Friedman, 2001), largely due to their short training time and robustness (Shwartz-Ziv and Armon, 2022).
We propose a radical change to how tabular classification is done.
We do not fit a new model from scratch to the training portion of a new dataset.
Instead, we replace this step by performing a single forward pass with a large Transformer that has been pre-trained to solve artificially generated classification tasks from a tabular dataset prior.
Our method builds on Prior-Data Fitted Networks (PFNs; Müller et al., 2022; see Section 2), which learn the training and prediction algorithm itself.
PFNs approximate Bayesian inference given any prior one can sample from and approximate the posterior predictive distribution (PPD) directly.
While inductive biases in NNs and GBDTs depend on them being efficient to implement (e.g., through
L 2 regularization, dropout (Srivastava et al., 2014) or limited tree-depth), in PFNs, one can simply design a dataset-generating algorithm that encodes the desired prior.
This fundamentally changes the way we can design learning algorithms.
We design a prior (see Section 4) based on Bayesian Neural Networks (BNNs; Neal 1996; Gal 2016) and Structural Causal Models (SCMs; Pearl 2009; Peters et al. 2017) to model complex feature dependencies and potential causal mechanisms underlying tabular data.
Our prior also takes ideas from Occam's razor: simpler SCMs and BNNs (with fewer parameters) have a higher likelihood.
Our prior is defined via parametric distributions, e.g., a log-scaled uniform distribution for the average number of nodes in data-generating SCMs.
The resulting PPD implicitly models uncertainty over all possible data-generating mechanisms, weighting them by their likelihood given the data and their prior probability.
Thus, the PPD corresponds to an infinitely large ensemble of data-generating mechanisms, i.e., instantiations of SCMs and BNNs.
We learn to approximate this complex PPD in a single forward-pass, requiring no cross-validation or model selection.
Our key contribution is to introduce the TabPFN (see Section 3), a single Transformer that has been pre-trained to approximate probabilistic inference for the novel prior above (described in more detail in Section 4) in a single forward pass, and has thus learned to solve novel small tabular classification tasks ( ≤ 1 000 training examples, ≤ 100 purely numerical features without missing values and ≤classes) in less than a second yielding state-of-the-art performance.
To substantiate this claim, we qualitatively and quantitatively analyze the behavior and performance of our TabPFN on different tasks and compare it to previous approaches for tabular classification on 18 small, numerical datasets (see Section 5).
Quantitatively, the TabPFN yields much better performance than any individual “base-level” classification algorithm, such as gradient-boosting via
XGBoost (Chen and Guestrin, 2016), LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018), and in less than a second yields performance competitive to what the best available
AutoML frameworks (Erickson et al., 2020; Feurer et al., 2021) achieve in one hour.
Our indepth qualitative analysis shows the TabPFN's predictions to be smooth and intuitive.
Yet, its errors are quite uncorrelated to the errors of existing approaches, allowing additional performance improvements by ensembling.
We also validate TabPFN’s performance on an additional 67 datasets from OpenML Vanschoren et al.
We expect the revolutionary character of our claims to be met with initial skepticism and thus opensource all our code and the pre-trained TabPFN for scrutinization by the community, along with a scikit-learn-like interface, a Colab notebook and two online demos.
Please see Section 8 for the links.
B ACKGROUND ON P RIOR -D ATA F ITTED N ETWORKS (PFN S )
First, we summarize how PFNs work; we refer to Müller et al.
for the full details.
The Posterior Predictive Distribution for Supervised Learning In the Bayesian framework for supervised learning, the prior defines a space of hypotheses Φ on the relationship of a set of inputs x to the output labels y .
Each hypothesis ϕ ∈ Φ can be seen as a mechanism that generates a data distribution from which we can draw samples forming a dataset.
For example, given a prior based on structural causal models, Φ is the space of structural causal models, a hypothesis ϕ is one specific
SCM, and a dataset comprises samples generated through this SCM.
In practice, a dataset comprises training data with observed labels and test data where labels are missing or held out to assess predictive performance.
The PPD for a test sample x test specifies the distribution of its label p ( ·| x test , D train ) , which is conditioned on the set of training samples D train :
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } .
The PPD can be obtained by integration over the space of hypotheses Φ , where the weight of a hypothesis ϕ ∈ Φ is determined by its prior probability p ( ϕ ) and the likelihood p ( D | ϕ ) of the data D given ϕ :
[Equation]
Synthetic Prior-fitting Prior-fitting is the training of a PFN to approximate the PPD and thus do
Bayesian prediction.
We implement it with a prior which is specified by a prior sampling scheme of the form p ( D ) =
E ϕ ∼ p
ϕ ) [ p ( D | ϕ )] , which first samples hypotheses (generating mechanisms) with ϕ ∼ p ( ϕ ) and then synthetic datasets with D ∼ p
( D | ϕ ) .
We repeatedly sample such synthetic datasets D :=
( x i , y i ) i ∈{ 1 ,...,n } and optimize the PFN's parameters θ to make predictions for
D test ⊂ D , conditioned on the rest of the dataset D train =
D \ D test .
The loss of the PFN training thus is the cross-entropy on held-out examples of synthetic datasets.
For a single test point { ( x test , y test ) }
=
D test , the training loss can be written as
[Equation]
As shown by Müller et al. (2022), minimizing this loss approximates the true Bayesian posterior predictive distribution.
We visualize this in Figure 1a and detail the full training setup in Algorithmin the appendix.
Crucially, this synthetic prior-fitting phase is performed only once for a given prior p ( D ) as part of algorithm development.
[Figure]
Real-World Inference During inference
, the trained model is applied to unseen real-world datasets.
For a novel dataset with training samples D train and test features x test , feeding ⟨ D train , x test ⟩ as an input to the model trained above yields the PPD q θ ( y | x test , D train ) in a single forward-pass.
The
PPD class probabilities are then used as predictions for our real-world task.
Thus, PFNs perform training and prediction in one step (similar to prediction with Gaussian Processes) and do not use gradient-based learning on data seen at inference time.
Architecture PFNs rely on a Transformer (Vaswani et al., 2017) that encodes each feature vector and label as a token, allowing token representations to attend to each other, as depicted in Figure 1b.
They accept a variable length training set D train of feature and label vectors (treated as a set-valued input to exploit permutation invariance) as well as a variable length query set of feature vectors x test =
{ x ( test, 1) , . . .
, x ( test,m ) } and return estimates of the PPD for each query.
Tabular Data Amongst other experiments, Müller et al.
(2022) demonstrated PPD approximation with PFNs on binary classification for tiny, balanced tabular datasets with 30 training examples.
Here, we substantially improve performance and scale up to small datasets with up to 1 000 data points and up to 10 classes, including imbalance.
We discuss detailed changes in Appendix C.2.6.
T HE T AB PFN: A PFN FITTED ON A NEW PRIOR FOR TABULAR DATA
Our TabPFN is a Prior-data Fitted Network (PFN, see Section 2) that is fitted on data sampled from a novel prior for tabular data we introduce in Section 4.
We modify the original PFN architecture (Müller et al., 2022) in two ways.
i)
We make slight modifications to the attention masks, yielding shorter inference times.
ii) Additionally, we enable our model to work on datasets with different numbers of features by zero-padding.
These architectural enhancements alongside our main contributions compared to Müller et al.
(2022) are detailed in Appendix E.2.
In the prior-fitting phase, we train the TabPFN once on samples from the prior described in Section 4.
To be more precise, we trained a 12-layer Transformer for 18 000 batches of 512 synthetically generated datasets each, which required a total of 20 hours on one machine with 8 GPUs (Nvidia
RTX 2080 Ti).
This yielded a single network that is used for all our evaluations.
While this training step is moderately expensive, it is done offline, in advance, and only once for the TabPFN, as part of our algorithm development.
The same TabPFN model is used for all experiments in this paper.
Full details about our TabPFN training are given in Appendix E.
During inference, the TabPFN approximates the PPD for our dataset prior, i.e., it approximates the marginal predictions across our spaces of SCMs and BNNs (see Section 4), including a bias towards simple and causal explanations for the data.
In our experiments, we present predictions for a single forward pass of our TabPFN, as well as predictions that ensemble 32 forward passes of datasets modified by a power transformation (applied with probability 0.5) and rotating the indices of feature columns and class labels (see Appendix C.2.6 for details).
z (1) X z (2)
X z (1) z
(2) z (3) z (4)
z y (a) A BNN z (4) z
(5) z (1) X z (2) z
(2) X z (3) z
z y (b)
An SCM z (3) z
(4) z (1) z (2) x (2) x (1) y z (3) z (4) x (1) x (2) z (2) x (3) z (1) y z (5) z
(6) z (4) x (1) x (2) z (5) z (3) y z
(1) z (2) (
[Figure]
SCMs sampled from the prior
A P RIOR FOR T ABULAR D ATA
The performance of our method crucially depends on the specification of a suitable prior, as the PFN approximates the PPD for this prior.
Section 4.1 outlines a fundamental technique for our prior: we use distributions instead of point-estimates for almost all of our prior’s hyperparameters.
Section 4.
motivates simplicity in our prior, while Sections 4.3 and 4.4 describe how we use Structural Causal
Models (SCMs) and Bayesian Neural Networks (BNNs) as fundamental mechanisms to generate diverse data in our prior.
Since our SCM and BNN priors only yield regression tasks, we show how to convert them to classification tasks in Section 4.5.
We describe additional refinements to our prior to reflect peculiarities of tabular data (correlated and categorical features, exponentially scaled data and missing values) better in Appendix C.2.
4. F UNDAMENTALLY P ROBABILISTIC M ODELS
Fitting a model typically requires finding suitable hyperparameters, e.g., the embedding size, number of layers and activation function for NNs.
Commonly, resource-intensive searches need to be employed to find suitable hyperparameters (Zoph and Le, 2017; Feurer and Hutter, 2019).
The result of these searches, though, is only a point estimate of the hyperparameter choice.
Ensembling over multiple architectures and hyperparameter settings can yield a rough approximation to a distribution over these hyperparameters and has been shown to improve performance (Zaidi et al., 2021; Wenzel et al., 2020).
This, however, scales linearly in cost with the number of choices considered.
In contrast, PFNs allow us to be fully Bayesian about our prior’s hyperparameters.
By defining a probability distribution over the space of hyperparameters in the prior, such as BNN architectures, the
PPD approximated by our TabPFN jointly integrates over this space and the respective model weights.
We extend this approach to a mixture not only over hyperparameters but distinct priors: we mix a
BNN and an SCM prior, each of which again entails a mixture of architectures and hyperparameters.
4. S IMPLICITY
We base our priors on a notion of simplicity, such as stated by Occam’s Razor or the Speed Prior (Schmidhuber, 2002).
When considering competing hypotheses, the simpler one is to be preferred.
Work in cognitive science has also uncovered this preference for simple explanations in human thinking (Wojtowicz and DeDeo, 2020).
Any notion of simplicity, however, depends on choosing a particular criterion that defines simplicity.
In the following, we introduce priors based on SCMs and
BNNs, in which we implement simplicity as graphs with few nodes and parameters.
4. SCM P RIOR
It has been demonstrated that causal knowledge can facilitate various ML tasks, including semisupervised learning, transfer learning and out-of-distribution generalization (Schölkopf et al., 2012;
Janzing, 2020; Rothenhäusler et al., 2018).
Tabular data often exhibits causal relationships between columns, and causal mechanisms have been shown to be a strong prior in human reasoning (Waldmann and Hagmayer, 2013; Wojtowicz and DeDeo, 2020).
Thus, we base our TabPFN prior on SCMs that model causal relationships (Pearl, 2009; Peters et al., 2017).
An SCM consists of a collection
Z :=
( { z 1 , . . . ,
z k } ) of structural assignments (called mechanisms): z i =
f i ( z PA G ( i ) , ϵ i ) , where
[Figure]
PA G ( i ) is the set of parents of the node i (its direct causes) in an underlying DAG G (the causal graph), f i is a (potentially non-linear) deterministic function and ϵ
i is a noise variable.
Causal relationships in G are represented by directed edges pointing from causes to effects and each mechanism z i is assigned to a node in G , as visualized in Figure 2.
Predictions based on ideas from causal reasoning Previous works have applied causal reasoning to predict observations on unseen data by using causal inference, a method which seeks to identify causal relations between the components of a system by the use of interventions and observational data (Pearl, 2010; Pearl and Mackenzie, 2018; Lin et al., 2021).
The predicted causal representations are then used to make observational predictions on novel samples or to provide explainability.
Most existing work focuses on determining a single causal graph to use for downstream prediction, which can be problematic since most kinds of SCMs are non-identifiable without interventional data, and the number of compatible DAGs explodes due to the combinatorial nature of the space of DAGs.
In contrast, our TabPFN considers a broad family of SCMs but without any causal guarantees.
We skip any explicit graph representation in our inference step and approximate the PPD directly.
Thus, we do not perform causal inference but solve the downstream prediction task directly.
This implicit assumption of SCM-like processes generating our data can be explained in Pearl's “ladder of causation”, an abstraction of inference categories, where each higher rung represents a more involved notion of inference (Pearl and Mackenzie, 2018).
At the lowest rung lies association, which includes most of ML.
The second rung considers predicting the effect of interventions, i.e., what happens when we influence features directly.
Our work can be considered as “rung 1.5”, similar to Kyono et al.
we do not perform causal reasoning, but make association-based predictions on observational data assuming SCMs model common datasets well.
In Figure 8 in Appendix B, we experimentally show that TabPFN’s predictions indeed align with simple SCM hypotheses.
Defining a prior based on causal models
To create a PFN prior based on SCMs, we have to define a sampling procedure that creates supervised learning tasks (i.e., datasets).
Here, each dataset is based on one randomly-sampled SCM (including the DAG structure and deterministic functions f i ).
Given an SCM, we sample a set z X of nodes in the causal graph G , one for each feature in our synthetic dataset, as well as one node z y from G .
These nodes are observed nodes: values of z X will be included in the set of features, while values from z y will act as targets.
For each such
SCM and list of nodes z X and z y , n samples are generated by sampling all noise variables in the
SCM n times, propagating these through the graph and retrieving the values at the nodes z X and z y for all n samples.
Figure 2b depicts an SCM with observed feature- and target-nodes in grey.
The resulting features and targets are correlated through the generating DAG structure.
This leads to features conditionally dependent through forward and backward causation, i.e., targets might be a cause or an effect of features.
In Figure 3, we compare samples generated by two distinct SCMs to actual datasets, demonstrating the diversity in the space of datasets our prior can model.
In this work, we instantiate a large subfamily of DAGs and deterministic functions f i to build SCMs described in Appendix C.1.
Since efficient sampling is the only requirement we have, the instantiated subfamily is very general, including multiple activation functions and noise distributions.
[Figure]
4. BNN P RIOR
We also consider a BNN prior as introduced by Müller et al.
and mix it with the SCM prior described above by randomly sampling datasets during PFN training from either one or the other prior with equal probability.
To sample a dataset from the BNN prior, we first sample an NN architecture and its weights.
Then, for each data point in the to-be-generated dataset, we sample an input x , feed it through the BNN with sampled noise variables and use the output y as a target (see Figure 2a).
An experimental ablation of this BNN prior to our final mixture of both SCM-based and BNN-based priors in Appendix B.4 demonstrates the strength of our novel SCM-based prior.
4. M ULTI - CLASS P REDICTION
So far, the described priors return scalar labels.
In order to generate synthetic classification labels for imbalanced multi-class datasets, we need to transform our scalar labels ˆ y to discrete class labels y .
We do so by splitting the values of ˆ y into intervals that map to class labels:
i)
We sample the number of classes N c ∼ p ( N c ) , where p ( N c ) is a distribution over integers.
ii)
We sample N c − 1 class bounds B i randomly from the set of continuous targets ˆ y .
iii)
We map each scalar label ˆ y i to the index of the unique interval that contains it: y i ← (cid:80)
j [ B j < ˆ y i ] , where [ · ] is the indicator function.
For example, with N c = 3 classes the bounds B c = {− 0 . 1 , 0 . 5 } would define three intervals { ( −∞ , − 0 .
1] , ( − 0 .
5] , (0 . 5 , ∞ ) } .
Any ˆ y i would be mapped to the label
if it is smaller than − 0 .
1 , to 1 if lies in ( − 0 .
5] and to 2 otherwise.
Finally, we shuffle the labels of classes, i.e. we remove the ordering of class labels w.r.t.
the ranges.
E XPERIMENTS 5. E VALUATION ON T OY P ROBLEMS
We first qualitatively compare our TabPFN to standard classifiers (without hyperparameter tuning) in Figure 4.
The top row shows the moons dataset with noise.
The TabPFN accurately models the decision boundary between samples; also, similar to Gaussian processes, uncertainties are large for points far from observed samples.
The second row shows the circles dataset with noise: the TabPFN accurately models the circle's shape with high confidence anywhere outside the region where samples are mixed.
The third row shows two classes and features from the iris dataset, while the fourth rows shows two classes and features from the wine dataset (both in scikit-learn (Pedregosa et al., 2011)); in both of these cases, the TabPFN makes intuitive, well-calibrated predictions.
5. E VALUATION ON T ABULAR ML T ASKS
Now, we turn to an empirical analysis of our method for real-world classification tasks.
We compare our method against state-of-the-art ML and AutoML methods for tabular classification.
1s 5s 30s 5min 1h 0 .0 .0 .0 .0 .0 .0 .
Given Time Budget
M ea n R O
C
AU C 1s 5s 30s 5min 1h
Given Time Budget
R O C
AU C
W i n s 1s 5s 30s 5min 1h
Given Time Budget
M ea n R O
C
AU C
R a nk
TabPFN
Auto-sklearn 2.
Autogluon
LightGBM
XGBoost
LogReg
Cocktail
Catboost
KNN
SAINT
As test datasets, we used all datasets from the curated open-source OpenML-CC benchmark suite (Bischl et al., 2021) that contain up to 2 000 samples ( 1 000 for the training split), 100 features and 10 classes.
The resulting set comprises 30 datasets.
We split these datasets into 18 datasets that contain only numerical features and no missing values, and 12 other datasets that contain categorical features and/or missing values.
In the main paper, in Figure 5 and Table 1, we limit our analysis to the case of numerical datasets without missing values, which we focussed the development of our TabPFN prior on.
Results on all of the 30 datasets are given in Appendix B.1 and still show strong aggregate performance for TabPFN, albeit not as strong as for the purely numerical case, due to generally worse performance of TabPFN on datasets with categorical features and/or missing values.
We focus on small datasets because (1) small datasets are often encountered in real-world applications (Dua and Graff, 2017), (2) existing DL methods are most limited in this domain (Grinsztajn et al., 2022) and (3) the TabPFN would be significantly more expensive to train and evaluate for larger datasets, a limitation detailed in Appendix A.
Baselines We compare against five standard ML methods and two state-of-the-art AutoML systems for tabular data.
As ML models we considered two simple and fast baselines, K-nearest-neighbors (KNN) and Logistic Regression (LogReg) .
Additionally, we considered three popular tree-based boosting methods, XGBoost (Chen et al., 2020), LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018).
For each ML model, we used 5 -fold cross-validation to evaluate randomly drawn hyperparameter configurations until a given budget was exhausted or 10 000 configurations were evaluated (for the search spaces, see Appendix F.2).
We then chose the best-performing hyperparameter configuration (maximum ROC AUC OVO) and refit on the whole training set.
Where necessary, we imputed missing values with the mean, one-hot or ordinal encoded categorical inputs, normalized features and passed categorical feature indicators.
As more complex but powerful baselines, we chose two state-of-the-art AutoML systems: AutoGluon (Erickson et al., 2020), which combines ML models including neural networks and tree-based models into a stacked ensemble, and Auto-sklearn 2.0 (Feurer et al., 2015; 2021) which uses Bayesian Optimization and combines the evaluated models into a weighted ensemble.
We note that previous works have found that DL baselines do not outperform or match the performance of GBDT or AutoML methods for small to medium-sized tabular data (< 10,000 samples; while matching GBDT performance on larger datasets) (Borisov et al., 2021;
Grinsztajn et al., 2022; Shwartz-Ziv and Armon, 2022).
Furthermore, DL methods such as TabNet, SAINT, Regularization Cocktails, Non-parametric Transformers (Arik and Pfister, 2021; Somepalli et al., 2021; Kadra et al., 2021; Kossen et al., 2021) are evaluated on much larger
[Table]
ROC AUC.
TabPFN n.e. is a faster version of our method, without ensembling, while TabPFN +
AutoGluon is an ensemble of TabPFN and AutoGluon.
Separate inference times in Table 2.
datasets and often use custom parameter tuning and preprocessing.
We still evaluate two prominent
DL methods: Regularization Cocktails 3 (Kadra et al., 2021) and SAINT (Somepalli et al., 2021).
Evaluation Protocol For each dataset and method, we evaluated 5 repetitions, each with a different random seed and train- and test split (50% train and 50% test samples; all methods used the same split given a seed).
To aggregate results across datasets, we report the ROC AUC (one-vs-one (OVO) for multi-class classification) average, ranks and wins including the 95% confidence interval and compare to the performance of the baselines with a budget of { 30 , 60 , 300 , 900 , 3600 } seconds.
Our TabPFN uses 32 data permutations for ensembling as described in Section 3; we also evaluate
TabPFN without permutations, which we label “TabPFN n.e. ”
in Table 1.
Results
We now present results for the 18 purely numerical datasets without missing values, detailed in Figure 5 and Table 1.
Figure 5 shows that TabPFN achieves a dramatically better tradeoff of accuracy and training speed than all the other methods: it makes predictions within less than a second on one GPU that tie with the performance of the best competitors (the AutoML systems) after training one hour, and that dominate the performance of tuned GBDT methods.
Unsurprisingly, the simple baselines ( LogReg , KNN ) already yield results with a small budget but perform worst overall.
GBDTs ( XGBoost , CatBoost , LightGBM ) perform better but are still outperformed by TabPFN and the state-of-the-art AutoML systems ( Auto-sklearn 2.0 , Autogluon ).
TabPFN is much faster than methods with comparable performance.
In the following, we compare the combined times for training and prediction (and tuning if applicable).
TabPFN n.e. requires 1.30s on a CPU and 0.05s on a GPU on average to predict for one dataset 5 , performing comparably to the strongest baselines at five minutes; it thus yields a 230 × speedup on CPU and a 5 700 × speedup using a GPU.
We ignore computational development costs of each method, see Appendix F.5 for why.
We would like to emphasize that the discussed results are aggregate results across datasets, and that no classification method, including TabPFN, performs best on all individual datasets.
Indeed, there do exist datasets for which TabPFN is outperformed even by default baselines.
Generally, TabPFN is less strong when categorical features or missing values are present.
Appendix B.1 presents results for all 30 test datasets from the OpenML-CC18 Benchmark, including 12 with categorical features and/or missing values.
Appendix B.5 presents a more detailed overview of the results for different kinds of datasets, including on an additional 149 validation datasets from OpenML, confirming TabPFN’s strong performance for purely numerical datasets.
We also show per dataset results for each of thedatasets (our 30 test datasets and the 149 validation datasets); these demonstrate that, while TabPFN generally does better for datasets with numerical features, there also exist several purely numerical datasets for which TabPFN does not perform better than the baselines, as well as categorical datasets for which it does clearly perform best.
OpenML-AutoML Benchmark
Additionally, we evaluated our TabPFN on the small ( ≤ 1training samples, 100 features, 10 classes) datasets of the OpenML-AutoML Benchmark, for which externally-validated performance numbers are available.
We use the setup provided by the OpenMLAutoML Benchmark, i.e.: metrics, official evaluation scripts and pre-released evaluations for an even wider range of AutoML baselines.
Using only an average of 4 . 4 seconds per dataset on a single
CPU, compared to 60 minutes for the baselines, TabPFN outperformed all baselines in terms of mean cross-entropy, accuracy and the OpenML Metric 6 .
We provide detailed results in Appendix B.2.
In-depth analysis of TabPFN predictions
We evaluated our model predictions in a plethora of ways to provide extra insights, such as the inductive biases of our method, see Appendix B.3.
We confirm that TabPFN learns to make predictions biased towards simple causal explanations, as detailed in B.3.1, while GBDT methods do not share this inductive bias.
We also evaluate our method's invariance to feature rotations and robustness to uninformative features in Appendix B.3.
and B.3.3.
In Figure 6, we observe that TabPFN is especially strong compared to baselines when datasets do not contain categorical features or missing values.
Ensembling We observe that TabPFN performs best on different datasets than our baselines, i.e., the per-dataset correlation of normalized ROC AUC scores between TabPFN and the strong baselines is lower than between the strong baselines, visualized in Figure 11 in our Appendix.
This is likely due to the novel inductive biases of our approach, which lead to distinct predictions.
Ensembling predictions is more effective when the considered methods make fewer correlated errors (Breimann, 2001), which encourages the use of more diverse strategies, making TabPFN an ideal candidate for ensembling with baseline methods (Wu et al., 2021).
Also, TabPFN is evaluated so quickly, that predictions are almost free, compared to the long runtimes of the baselines.
To demonstrate its potential for ensembling, we include an entry "TabPFN + AutoGluon" in Table 1, generated by averaging the predictions of TabPFN and AutoGluon ; this strongly outperforms all other methods.
Model Generalization
The PFN architecture accepts datasets of any length as input.
However, when training our model in the synthetic prior-fitting phase, we limited synthetic data to a maximum size of 1 024 , as prior-fitting becomes more expensive with larger datasets.
Nevertheless, we wondered:
Would TabPFN generalize to larger training set sizes that were never seen during training?
To test this, we used a collection of 18 datasets from the OpenML-AutoML Benchmark (see Table 11) from which we selected 10 000 samples each.
We evaluated TabPFN on up to 5 000 training samples, using 5 random data splits and 5000 test samples.
Surprisingly, our models generalize beyond sample sizes seen during training, as shown in Figure 10 in Appendix B.1.
C ONCLUSIONS & F UTURE W ORK
We have shown how a single Transformer, the TabPFN, can be trained to do the work of a full
AutoML framework for tabular data and can yield predictions in 0.4 seconds that are competitive with the performance that the best available AutoML frameworks achieve in 5 to 60 minutes.
This slashes the computational expense of AutoML, enabling affordable, green, AutoML.
The TabPFN still has important limitations: the underlying Transformer architecture only scales to small datasets as detailed in Appendix A; our evaluations focused on classification datasets with only up to 1 000 training samples, 100 purely numerical features without missing values andclasses, which motivates work on (1) scaling up to large datasets.
Our in-depth analysis of the inductive biases of TabPFN (see Appendix B.3), points towards extensions for (2) improved handling of categorical features, (3) missing values and (4) robustness to unimportant features.
Also, our work motivates a multitude of exciting follow-ups regarding (5) integration of TabPFN into existing
AutoML frameworks; (6) ensembling to continue making improvements given more time; (7) datasetdependent choices of the prior; (8) generalizations to non-tabular data and (9) regression tasks, as well as studies of the TabPFN regarding the dimensions of trustworthy AI (e.g., (10) out-of-distribution robustness; (11) algorithmic fairness, (12) robustness to adversarial examples, and (13) explainability).
The almost instant state-of-the-art predictions of TabPFN are also likely to give rise to (14) novel exploratory data analysis methods, (15) novel feature engineering methods and (16) novel active learning methods.
Finally, our advances in causal reasoning warrant follow-ups on (17) approximating the effects of interventions and counterfactuals considering a distribution of SCMs.
E THICS S TATEMENTS
In terms of broader societal impact of this work, we do not see any foreseeable strongly negative impacts.
However, this paper could positively impact the carbon footprint and accessibility of learning algorithms.
The computations required for machine learning research have been doubling every few months, resulting in a large carbon footprint (Schwartz et al., 2020).
Moreover, the financial cost of the computations can make it difficult for academics, students, and researchers to apply these methods.
The decreased computational time shown by TabPFN translates to reductions in CO2 emissions and cost, making it available to an audience that does not have access to larger scale computing.
As the TabPFN provides a highly portable and convenient way of building new classifiers that work in real-time, it is likely to increase the pervasiveness of machine learning even further.
While this can have many positive effects on society, such as better personalized healthcare, increased customer satisfaction, efficiency of processes, etc, it will also be crucial to study and improve the TabPFN under the lens of the many dimensions of trustworthy AI other than computational sustainability, such as algorithmic fairness, robustness to adversarial examples, explainability, and auditability.
We hope that its foundation in causal models and simplicity will allow possible avenues for work along these lines.
R EPRODUCIBILITY
Code release In an effort to ensure reproducibility, we release code alongside our pre-trained
TabPFN and notebooks to reproduce our experiments at https://github.com/automl/
TabPFN .
Application to public benchmarks In our work, we evaluate TabPFN to publicly available benchmarks: the OpenML-CC18 Benchmark and the OpenML-AutoML Benchmark.
This ensures, that dataset choices are not cherry-picked to our method.
Also, for the OpenML-AutoML Benchmark, we use official baseline results and evaluate our method using the evaluation scripts published for this benchmark.
Availability of datasets All datasets used in our experiments are freely available at OpenML.
org (Vanschoren et al., 2014), with downloading procedures included in the submission.
Further details on the datasets used can be found in Section F.3.
Online resources We created a Colab notebook, that lets you interact with our scikit-learn interface at https://colab.research.google.com/drive/1J0l1AtMV_
H1KQ7IRbgJje5hMhKHczH7-?usp=sharing .
We created another Colab, where our evaluation and plots on 179 test and validation datasets can be reproduced easily.
https://colab.research.google.com/drive/ 1yUGaAf3D7RSyO5Jc4PYXSVbtaUAozh6S
We also created two demos.
One to experiment with the TabPFNs predictions ( https: //huggingface.co/spaces/TabPFN/TabPFNPrediction ) and one to check crossvalidation ROC AUC scores on new datasets ( https://huggingface.co/spaces/TabPFN/
TabPFNEvaluation ).
Both of them run on a weak CPU, thus it can require a little bit of time.
Details of training procedures for TabPFN and baselines Details shared in the training procedure of all our Transformer models can be found in Appendix F.
An overview of the hyperparameters used for running the TabPFN and our baselines can be found in Tables 5 and 6, respectively.
A CKNOWLEDGMENTS
Robert Bosch GmbH is acknowledged for financial support.
This research was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG, and TAILOR, a project funded by EU Horizonresearch, and innovation programme under GA No 952215.
We acknowledge funding through the
European Research Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765).
Funded by the European Union.
Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC.
Neither the European
Union nor the ERC can be held responsible for them
A L IMITATIONS
The runtime and memory usage of the Transformer-based PFN architecture used in this work scales quadratically with the number of inputs, i.e., training samples passed.
Thus, inference on larger sequences (> 100 000 ) is hard on current consumer GPUs.
A growing number of methods seek to tackle this issue and report similar performances while scaling linearly with the number of inputs (Zaheer et al., 2020; Beltagy et al., 2020).
These methods can be integrated into the PFN architecture and thus into the TabPFN.
Furthermore, in our experiments we limit the number of features toand the number of classes to 10 as described in Section 5.
While this choice is flexible, the precise
TabPFN that we fitted cannot work with datasets that go beyond these limits.
We also focused the development of TabPFN to purely numerical datasets without missing values, and while they can be applied to datasets with categorical features and/or missing values, their performance is generally worse.
We hope to tackle this problem in future versions of TabPFN by a modified architecture and prior.
Finally, we did not consider the existence of many uninformative features in our prior, leading to performance degradation when such features are added; we hope to address this issue in future versions of TabPFN.
While baseline models (except Gaussian Processes) fit and predict in separate steps, TabPFN performs both at the same time.
Thus, baseline models are often faster than TabPFN in terms of pure inference time (see Table 2).
B A DDITIONAL R ESULTS
B.
D ETAILED T ABULAR R ESULTS
[Figure]
TabPFN
Autosklearn
Autogluon
Catboost
XGB
LGBM
Reg.
Cocktail
[Figure]
In Figure 6, we explore how the kind of dataset evaluated affects the performance of TabPFN, compared to our baselines.
We find that TabPFN performs much better when no categorical features are present.
We also find that TabPFN performs better, when no missing values are present in the data.
This warrants an extension of our prior in future work, to make it more customized towards categorical and missing data.
Our method seems to work comparably well for binary and multi-class problems.
In addition to the results in the main paper in Section 5.2, we report detailed results on all 30 datasets in OpenML-CC18, which are small enough, but might include categorical features and missing values.
We show performance over time in Figure 7 and a wide range of performance values and per dataset results with a 1 hour time limit in Table 2.
1s 5s 30s 5min 1h 0 .0 .0 .0 .0 .0 .0 .
Given Time Budget
M ea n R O
C
AU C 1s 5s 30s 5min 1h
Given Time Budget
R O C
AU C
W i n s 1s 5s 30s 5min 1h
Given Time Budget
M ea n R O
C
AU C
R a nk
TabPFN
KNN
Reg.
Cocktails
Log
Catboost
SAINT
LightGBM
XGBoost
Autogluon
[Figure]
B. R ESULTS ON THE O PEN ML-A UTO ML B ENCHMARK
We evaluate TabPFN using the official benchmarking scripts, 8 datasets, splits and baselines results of the OpenML-AutoML Benchmark.
The full list of 5 datasets can be found in Table 10.
We note that these datasets are not disjoint from our evaluation datasets; in fact 3 of these (“credit-g”, “vehicle”, and “blood-transfusion-service-center”) were included in our evaluation datasets from the OpenML-CC18 Benchmark as well, while one dataset (“Australian”) was included in our list of 150 meta-validation datasets.
The evaluation using another set of benchmarking scripts with previously published train-test splits and baseline results, however, helps to confirm that: (1) Our baselines are well tuned and not outperformed by another method in the extensive OpenML-AutoML
Benchmark; (2) the runtimes of TabPFN are reproducible in a controlled environment provided by the OpenML-AutoML Benchmark; and (3) TabPFN is not overfit to datasplits or our evaluation metric.
While we used a 50-50 train-test split with 5 iterations for our experiments in Table 3, the
OpenML-AutoML Benchmark uses a 10-fold cross-validation, which results in a 90-10 splits withiterations.
Thus in the OpenML-AutoML Benchmark, all methods use more training samples than in our experiments on the OpenML-CC18 Benchmark, which leads to slightly stronger results.
B. I N - DEPTH ANALYSIS OF MODEL BIASES
Previous work by Grinsztajn et al.
(2022) empirically investigates the inductive biases of tree-based and deep-learning models.
They identify three challenges in developing tabular-specific models.
Models must be (1) able to fit irregular functions; (2) robust to uninformative features; and (3) preserve the orientation of the data.
We perform and extend this analysis in the following section.
We consider three model types in our analyses: TabPFN, GBDTs (LightGBM) and NNs (Standard Sklearn Multi
Layer Perceptron with a hidden dimensionality of 100).
We do not seek to make absolute comparisons between these model types, which would warrant tuning of our baselines, but only seek to investigate their qualitative behavior in the following experiments.
B.3. F ITTING IRREGULAR PATTERNS
We believe GBDT methods tend to learn non-smooth and irregular patterns in the targets, while MLPs learn smooth, low-frequency functions, as can be seen in Figure 4.
TabPFN learns target functions that are rather smooth, as suggested in Figure 4.
This is due to our model’s prior, which prefers
[Table]
simple SCMs as explanations and thus less irregular decision planes.
We note, however, that when many training samples are provided, TabPFN also fits more complex functions.
The tradeoff between complexity and number of training samples is explored in Figure 8.
Here, we evaluate the training set cross-entropy loss on synthetic data generated from random SCMs.
The number of training samples and the complexity of the generated data (number of hidden units in the data generating graph) is varied.
B.3. R OBUSTNESS TO UNINFORMATIVE FEATURES
Tabular datasets contain a large fraction of uninformative features (Grinsztajn et al., 2022).
To evaluate robustness to uninformative features, we add an increasingly large fraction of uninformative features to our data and show results in Figure 9.
Uninformative features are generated by copying existing features and shuffling their values randomly between samples.
We find that TabPFN and MLPs are less robust to uninformative features than LightGBM.
TabPFN could be adapted by including more uninformative features in the used prior.
In a second experiment we drop an increasingly large fraction of features.
We drop these features according to feature importance (ranked by a Random
Forest), first removing least informative features.
We show results in Figure 9 (middle) and observe that the classification accuracy of a TabPFN and GBDT is not much affected by removing up to
[Table]
MeanAcc.
0.793 ± .
0.763 ± .
0.775 ± .
0.763 ± .
0.761 ± .
0.784 ± .
0.794 ± .033 MeanCE 0.454 ± .
0.537 ± .
0.502 ± .
0.545 ± .
0.499 ± .
0.73 ± .
0.449 ± .
Meantime(s)      4.374 (CPU)
Complexity
N u m b e r
o f s a m p l e s
Log.
Regr.
Complexity
LightGBM
Complexity
Tabular PFN 0.
30% of the features, but constantly diminishes.
The MLP, which is less not robust to uninformative features, performs even better when the 20% least informative features are removed.
We use our testing tasks from the OpenML CC-18 Benchmark, and, to simplify analyses, we drop multiclass datasets and datasets that contain more than 50 features (as adding 100% more features to a dataset with more than 50 features yields more than 100 features, which TabPFN cannot handle).
B.3. I NVARIANCE TO FEATURE ROTATION
Each feature of a tabular dataset typically carries meaning individually, as expressed by column names, such as age or sex.
A learning algorithm is rotationally invariant in the sense of Ng (2004), if it is left unchanged when a rotation (unitary) matrix is applied to the features of both the training and testing set, i.e. when features are mixed.
To remove uninformative features under a feature rotation, an algorithm first has to restore the original orientation of the features, and then select informative ones.
A rotationally invariant algorithm discards the data orientation and thus has to restore the original orientation internally.
Thus, Ng (2004) shows that any rotationally invariant learning algorithm has a worst-case sample complexity that grows at least linearly in the number of irrelevant features.
[Figure]
[Figure] [Figure]
Method
R O
C
AU
C
False
True
[Figure]
lgbmlptabpfn
Percentage
[Figure]
features dropped
[Figure] [Figure]
[Figure]
lgb mlp tabpfn
[Figure]
Figure 9 (left) shows the change in test ROC AUC when randomly rotating our datasets, and confirms that only MLPs are rotationally invariant.
GBDT methods are highly sensitive to rotations, while
TabPFN is less sensitive, but still performs better when no rotations are applied.
The theoretical results by Ng (2004) and empirical results by Grinsztajn et al.
(2022) imply, that
TabPFN's diminishing performance when uninformative features are added is linked to it's relative rotation invariance.
Adjusting the prior to include more uninformative features could address these results.
We use datasets from our testing tasks from the OpenML CC-18 Benchmark with numerical features only, since rotating categorical datasets is problematic, as some GBDT classifiers treat categorical variables distinctly (e.g. generating embeddings per category).
[Figure]
ON THE SELECTION OF PRIOR MODELS
We perform ablation experiments for a prior based solely on BNNs, SCMs and a mix of both using a hyperparameter to control the sampling likelihood during prior-fitting.
The PFNs for each prior are fitted using less compute than in our final experiments, thus their scores generally are slightly worse.
The BNN prior, similar to the one used in Müller et al.
(2022), provides diminished performance compared to the priors based on SCMs.
Additionally, we can see that mixing BNN and SCM prior does not seem to make a big difference on our test set compared to a pure SCM prior.
[Table]
B. E XTENDED ANALYSIS ON A LARGER BENCHMARK OF DATASETS
We now provide an extended benchmark where we include an analysis on the additional 149 validation datasets 9 (listed in Table 8) in order to assess the generality of our results and to better understand
[Figure]
C
AU
C
M ea n
[Figure]
and weaknesses comparisons with default random forests ( support vector machines (SVMs) default XGBoost etc.
with their tuned versions one hour evaluation used 5 splits and followed the experimental setup described in
Figure 12 shows results for our 30 test datasets (separated into the 18 purely numerical datasets without missing values we focus on in the main paper, and the 12 remaining ones), and Figureshows results for the 149 validation datasets (again split into numerical datasets without missing values and the remainder).
These figures demonstrate that on the purely numerical datasets without missing values, TabPFN Pareto-dominates all other methods in aggregate results across datasets, in terms of tradeoffs between ROC and time spent.
For categorical datasets and datasets with missing values, it still performs well, but not nearly as well as for the numerical case without missing values we targeted.
We now also use statistical tests.
We note that when evaluating a new classifier, performance on one dataset is just one data point.
In order to apply statistical tests, we need to perform analyses across datasets.
The standard method for this are critical difference diagrams as introduced by Demšar
[Figure]
Figure 14 performs these statistical tests to compare methods in two regimes: fast runs that require up to 30 seconds and tuned runs that require up to an hour.
We note again that TabPFN is much stronger on purely numerical datasets without missing features.
In this case, for the short runs, it statistically significantly outperforms all other baselines.
For long runs, it also statistically significantly outperforms all other methods except the AutoML frameworks in 1-vs-1 comparisons; however, this is not visible in the figure because of the multiple testing correction applied due to the multiple pairwise tests. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
While the analysis above aggregates across datasets, variation across datasets is large and TabPFN by no means works best on all datasets.
All classifiers have worst and best cases, and we believe that it is important to complement our reports of TabPFN’s strong aggregate performance with the examples we encountered for which the current version does not do well.
We do show results for all datasets one-by-one at https://github.com/automl/TabPFNResults/ blob/main/individual_plots.pdf .
The dataset with the worst relative performance for TabPFN is collins , with results shown in
Figure 15.
While most other methods obtain a ROC AUC close to, or identical to 1.0, TabPFN only achieves around 98%.
We found this to be caused by uninformative features, which we have already shown to affect TabPFN’s performance negatively in Section B.3.2.
With just the 5 most important features (as judged by a random forest), TabPFN also achieves an accuracy of 1.0.
(In the aggregate analysis above, we of course count the original, poor performance of TabPFN, to not cheat.)
TabPFN also works poorly on purely categorical datasets (e.g., dataset sensory , Figure 16a) or with many missing values (e.g., dataset meta
Figure
[Figure]
TabPFN works best for purely numerical datasets, with no missing values, e.g., datasets Touch2 and vehicle , see Figure 17.
TabPFN still works well in several datasets where hyperparameter optimization does not help in the baselines (maybe due to overfitting), e.g. Pizza-cutter1 and arsenic-female-bladder ,
[Figure]
see Figure 18.
We attribute this to being Bayesian, having meta-learned not to overfit on small datasets with a simplicity prior using principles from causality.
[Figure]
It is not the case, however, that TabPFN works great on all numerical datasets without missing values and poorly on all categorical datasets.
For example, TabPFN works poorly on the numerical dataset pm10 and great on the categorical dataset monks-problem2 , see Figure 19.
It is therefore necessary to look beyond individual datasets to obtain the entire picture.
C D ETAILS OF THE T AB PFN P RIOR C. SCM P RIOR
The Sampling Algorithm We instantiate a subfamily of DAGs that can be efficiently sampled from by starting with a MLP architecture and dropping weights from it.
That is, to sample a dataset with k features and n samples from our prior we perform the following steps for each dataset:
(1) We sample the number of MLP layers l ∼ p ( l ) and nodes h ∼ p ( h ) and sample a graph
G
( Z , E ) structured like an l -layered MLP with hidden size h .
(2) We sample weights for each Edge E ij as W i,j ∼ p w ( · ) .
(3) We drop a random set of edges e ∈ E to yield a random DAG.
(4) We sample a set of k feature nodes N x and a label node N y from the nodes Z .
We sample the noise distributions p ( ϵ ) ∼ p ( p ( ϵ )) from a meta-distribution.
This yields an SCM, with all f i ’s instantiated as random affine mappings followed by an activation.
Each z i corresponds to a sparsely connected neuron in the MLP.
[Figure]
With the above parameters fixed, we perform the following steps for each member of the dataset:
(1) We sample noise variables
ϵ i from their specific distributions.
(2) We compute the value of all z ∈ Z with z i = a (( (cid:80) j ∈ PA G ( i ) E ij z j )
+ ϵ i ) .
(3) We retrieve the values at the feature nodes N x
and the output node N y and return them.
We sample one activation function a per dataset from { Tanh, LeakyReLU, ELU, Identity } (Nair and Hinton, 2010).
The sampling scheme for the number of layers p ( l ) and nodes p ( h ) is designed to follow a discretized noisy log-normal distribution, p ( ϵ ) is a noisy log-normal distribution and the dropout rate follows a beta distribution.
The full information can be found in Table 5.
C. T ABULAR D ATA R EFINEMENTS
Tabular datasets comprise a range of peculiarities, e.g. feature types can be numerical, ordinal, or categorical and feature values can be missing, leading to sparse features.
We seek to reflect these peculiarities in the design of our prior as described in the following sections
. C.2. P REPROCESSING
During prior-fitting, input data is normalized to zero mean and unit variance, and we apply the same step when evaluating on real data.
Since tabular data frequently contains exponentially scaled data, which might not be present during prior-fitting we apply power scaling during inference (Yeo and
Johnson, 2000).
Thus, during inference on real tabular datasets the features more closely match those seen during prior-fitting.
We use only training samples for calculating z-statistics, power transforms and all other preprocessing.
We take this preprocessing time into account when reporting the inference time of our method.
C
. C ORRELATED F EATURES
Feature correlation in tabular data varies between datasets and ranges from independent to highly correlated.
This poses problems to classical deep learning methods (Borisov et al., 2021).
When considering a large space of SCMs, correlated features of varying degrees naturally arise in our priors.
Furthermore, in real-world tabular data, the ordering of features is often unstructured, however adjacent features are often more highly correlated than others.
We use “Blockwise feature sampling” to reflect the correlation structure between ordered features.
Our generation method of SCMs naturally provides a way to do this.
The first step in generating our SCMs is generating a unidirectional layered network structure in which nodes in one layer can only receive inputs from the preceding layer.
Thus, features in the same layer tend to be more highly correlated.
We use this by sampling adjacent nodes in the layered network structure in blocks and using these ordered blocks in our set of features.
In
Figure 20, we visualize the correlations of such a generated dataset (right) and compare them to a real-world dataset (left), demonstrating that our prior yields correlation structures similar to those of real datasets.
C.2. G ENERATING IRREGULAR FUNCTIONS
In real-world data, some features are consistently more important than others.
While a random network weight initialization leads to slightly different feature importances, the average effect of input features regresses to the mean when the hidden dimensionality increases.
We amplify differences by sampling a weight parameter for each input feature and multiplying all outgoing weights by this factor.
In the prior, we randomly sparsify connections of the graph.
Thus hidden variables and the output node are influenced by fewer parameters, yielding more irregular patterns, as a larger number of parameters once again regresses to the mean.
We also extend sparsification to blocks of variables, leading to some groups of variables interacting more strongly.
We also extend the way noise variables are sampled.
Instead of sampling Gaussian noise at each node from the same distribution, we first sample separate noise means and standard deviations for each node and then sample from this distribution.
Also, we generate non-uniformly distributed input data x, as observed in real-world data: We sample the input variables x (which are propagated through our network), from a mix of distributions, namely the Gaussian, Zipfian and Multivariate Distribution.
C.2. N AN H ANDLING
We do not have special nan handling built into our model.
We replace nan values with zero at test time.
C.2. C ATEGORICAL F EATURES
Tabular data often includes not only numeric features but also discrete categorical ones.
While categorical features should technically not be ordered, in practice, they sometimes are, i.e., the categories represent binned degrees of some underlying variable.
We introduce categorical features by picking a random fraction p cat (a hyperparameter) of categorical features per dataset.
Analogous to transforming numeric class labels to discrete multiclass labels, we convert dense features to discrete ones.
Also analogous to multiclass labels, we pick a shuffling fraction of categorical features p scat where we reshuffle categories.
For details, see Section 4.5.
During prior-fitting, we use a probability for categorical features of 20% .
C.2. D IFFERENCES TO P RIOR WORK ON PFN S FOR T ABULAR D ATA
Prior work by Müller et al.
(2022) has demonstrated tabular data classification using PFNs, but was limited to 30 training samples, balanced binary-classification and 60 features.
Here we summarize the most important changes, to this prior work:
i)
The PFNs for tabular data described by Müller et al.
(2022) can only handle balanced binary datasets.
In Section 4.5, we show how to extend the prior to handle imbalanced classes and multi-class classification problems.
ii) We studied pre-processing techniques for the TabPFN, which was not done at all before.
This includes outlier removal and power scaling during inference (Yeo and Johnson, 2000).
During training, we also rotate feature indices (based on the assumptions that it should not be informative whether a feature is listed in the i -th or the j -th column, but that features may be listed in groups and that there may be more interactions between features with similar indices 10 ) and class labels (based on the same assumptions).
iii)
We construct ensembles over multiple pre-processings.
Specifically, ensemble members differ in the feature column rotation, the class label rotation, and the use of a power transform.
We only include each combination of the k possible feature column rotations (for k features), j possible class label rotations (for j -ary classification problems) andchoices of transformation (power transform, none).
Even if a number of ensemble members is specified that is larger than 2 kj , we only use 2 kj ensemble members.
iv)
We changed the transformer architecture to make it faster.
We shrank attention matrix sizes from ( n + m ) 2 to n 2 + n ∗ m , for n training points and m inference points.
v) We introduce a novel SCM prior.
We compare the SCM, the improved BNN prior (which is more heavily based on the BNN setup of C.2.6) and SCM + BNN in Table 4.
This improvesperformance by 2% in this smaller scale setup, which is a bigger difference than between the performance of the final TabPFN and all baselines besides KNN and SAINT.
D D ETAILS OF THE P RIOR -D ATA F ITTED N ETWORK A LGORITHM
Algorithm 1 describes the training method proposed by Müller et al.
(2022) for PFNs.
Algorithm 1: Prior-fitting of a PFN (Müller et al., 2022)
Input : A prior distribution over datasets p ( D ) , from which samples can be drawn and the number of samples K to draw
Output : A model q θ that will approximate the PPD
Initialize the neural network q
; for j ← 1 to K do end
E S ETUP OF OUR METHOD E. T RANSFORMER H YPERPARAMETERS
We considered only PFN Transformers with 12 layers, embeddings size 512 , hidden size 1024 in feed-forward layers, and 4 -head attention.
We used the Adam optimizer (Kingma and Ba, 2015) with linear-warmup and cosine annealing (Loshchilov and Hutter, 2017).
For each training we tested a set of 3 learning rates, { . 001 , .
0001 } , and used the one with the lowest final training loss.
The resulting model contains 25.82 M parameters.
E. PFN A RCHITECTURE A DAPTATIONS
Attention Adaption
The original PFN architecture (Müller et al., 2022) uses a single multi-head selfattention module (Vaswani et al., 2017) to compute the attention between all the training examples, as well as, the attention from validation examples to training examples.
We replaced this, with two modules that share weights, one which computes self-attention among the training examples and the other that only compute cross-attention from validation examples to training examples.
Conceptually, this is equivalent to the original architecture, except that we’re using a slightly different self-attention mask than the original architecture, which allowed all examples to attend to itself (the diagonal is 1), as in this example:
[Equation]
For validation examples, we remove the attention to themselves.
In terms of the example above:
[Equation]
Information about the state of the current position does still flow through the residual branch, though.
Flexible Encoder Datasets have unequal numbers of input dimensions (features), while PFNs use an encoder layer that accepts fixed dimensional inputs.
Here we explain how datasets with different numbers of dimensions can be modelled with a single PFN: We draw the number of dimensions of a dataset during training uniformly at random up to 100 .
Our encoder changes to accomodate this training and inference with different numbers of features by zero-padding datasets where the number of features k is smaller than the maximum number of features K and scaling these features by Kk , s.t.
the magnitude stays the same.
E. T AB PFN T RAINING
We trained our final model for 18 000 steps with a batch size of 512 datasets.
That is our TabPFN is trained on 9 216 000 synthetically generated datasets.
This training takes 20 hours on 8 GPUs (Nvidia
RTX 2080 Ti).
Each dataset had a fixed size of 1024 and we split it into training and validation uniformly at random.
We generally saw that learning curves tended to flatten after around 10 million datasets and were generally very noisy.
Likely, this is because our prior generates a wide variety of different datasets.
E.
[Equation]
The hyperparameters of our prior were chosen based on simplicity and our observations on the validation datasets (such as their class distributions or feature correlation strengths).
Also, during algorithm development, we evaluated our models on this set of datasets to decide if our developed methods were correct and working.
Since our prior hyperparameters specify distributions and not definite values, they can be chosen over a wide range and resemble the intervals chosen for a random hyperparameter search.
The prior distributions we used are given in Table 5.
F D ETAILS FOR T ABULAR E XPERIMENTS
Here we provide additional details for the experiments conducted in Section 5 in the main paper.
F. H ARDWARE S ETUP
All evaluations, including the baselines, ran on a compute cluster equipped with Intel(R)
Xeon(R)
Gold 6242 CPU @ 2.80GHz using 1 CPU with up to 6 GB RAM.
For evaluation using our TabPFN, we additionally use an RTX 2080 Ti.
[Table]
B ASELINES
We provide the search space used to tune our baselines in Table 6.
For CatBoost and XGBoost , we used the same ranges as Shwartz-Ziv and Armon (2022) with the following exception: For CatBoost we removed the hyperparameter max_size since we could not find it in the official documentation.
To be maximally fair to XGBoost, we also tried the search space of quadruple Kaggle grandmaster
Bojan Tunguz (Tunguz, 2022), which we adapted slightly by using softmax instead of logistic, as we are in the multi-class setting.
XGBoost with this search space performed worse for all considered time budgets than the search space by Shwartz-Ziv and Armon (2022).
The search spaces for the
KNN, GP and Logistic Regression baselines were designed from scratch and we used the respective implementation from scikit-learn (Pedregosa et al., 2011).
For CatBoost and AutoSklearn , we pass the position of categorical features to the classifier ( AutoGluon automatically detects categorical feature columns).
We normalize inputs for Logistic Regression, GP and KNN to the range [0, 1] using MinMax Scaling.
F. U SED D ATASETS
To construct and evaluate our method, we used the following four sets of datasets.
First, our meta-test set (see Table 7) comprises all datasets in the OpenML-CC18 benchmark suite (Bischl et al., 2021)(available at OpenML.org ) with at most 2 000 samples, 100 features and 10 classes, which leaves us with 30 datasets that represent small, tabular datasets.
Second, our meta-validation set (see Tables 8 and 9) comprises 150 datasets from OpenML.org (Vanschoren et al., 2014).
For this, we considered all datasets on OpenML.org and applied the following filtering procedure: We dropped all datasets that are in the meta-test set and all datasets with more than 2 000 samples, 100 features or 10 classes.
We also manually checked for overlaps and removed datasets where the number of features, classes and samples was identical to a dataset in the meta-test set.
Furthermore, we manually dropped FOREX (since it is a time series dataset) and artificiallycreated datasets, such as the Univ and Friedman datasets.
The remaining meta-validation set then contains 150 datasets.
This meta-validation set was used to guide the development of our prior hyperparameters as described in Appendix E.4.
[Table]
b a l a n c es c a l e m f e a t - f o u r i e r b r e a s t - w m f e a t - k a r hun e n m f e a t - m o r p
h
o l
o g
i c a l m f e a t - z e r n i k
e c m c c r e d
i t - a pp r o v a l c r e d i t - g d i a b e t e s t
i c - t a c - t o e v e h
i c
l e e
u c a l y p t u s a n a l c a t d a t a _ a u t h
o r s h
i p a n a l c a t d a t a _ d m f t p c
4 p c 3 k c 2 p cb a n k n o t e - a u t h e n t
i c a t i o n b l oo d - t r a n s f u s i o n - s e r v
i c
e - c e n t e r il pd q s a r - b i o d e g w db c c y li n d e rb a n d s d r e ss e ss a l e s
M i c e P r o t e i n c a r s t ee l - p l a t e s - f a u l t c li m a t em o d e l - s i m u l a t
i o n - c r a s h e s
[Figure] [Figure]
for each dataset.
Ensemble weights are averaged across 5 splits for each dataset in the OpenML-CC18 Benchmark after one hour of training.
Third, a subset of 5 datasets from the OpenML-AutoML Benchmark, comprises all datasets from the
OpenML-AutoML Benchmark with at most 1 111 samples, 100 features and 10 classes.
This is given in Table 10.
Due to the 10-fold cross-validation in the OpenML-AutoML Benchmark this is identical to the setup for our meta-test and meta-validation datasets above, where we used up to 2 000 samples split 50-50 into training and test.
Fourth, our meta-generalization set, described in Appendix F.4, comprises 18 larger datasets from the
OpenML AutoML Benchmark.
F. M ODEL G ENERALIZATION
For testing the TabPFN performance on longer sequences, as described in Section 10, we used a set of 18 datasets from the OpenML AutoML Benchmark that contain at least 10 000 samples.
The list of datasets used can be found in Table 11.
For this evaluation, datasets with more than 100 features are limited to the first 100 features.
When more than 10 classes are contained in the datasets, samples with any but the first 10 classes are discarded.
F. D ETAILS ON T IME C OMPARISONS
Time comparisons refer to combined fitting, tuning and prediction; see Table 2 for the times split into tuning/fitting and prediction.
The time taken for each baseline and TabPFN does not include the one-time cost of development of each method.
Thus, for AutoML baselines meta-learning cost was not included (e.g. Auto-Sklearn pipelines meta-learning, which involved running hyper-parameter search for 24 hours on 140 datasets (= 3360 CPU hours) (Feurer et al., 2021)).
For GBDT methods the manual time that went into defining suitable hyperparameter spaces and the manual crafting of algorithms that perform well on tabular datasets is hard to measure and was not included.
For TabPFN the prior-fitting phase (which is part of our algorithm development: i.e. developing ideas, writing code, trying ideas) is not included.
This is fair to all methods, as these costs are not on the user side and are amortized over time.
[Table]
[Figure]
[Table]
disclosure
[Table]
774 bodyfat      778 cleveland      786 triazines      788 disclosure_x_tampered      795 cpu      796 cholesterol      798 chscase_funds      801 pbcseq      802 pbc      810 rmftsa_ctoarrivals      811 chscase_vine       814 chatfield_       820 boston_corrected      825 sensory      826 disclosure_x_noise      827 autoMpg      831 kdd_el_nino-small      839 autoHorse      840 stock      841 breastTumor      844 analcatdata_gsssexsurvey      852 boston      853 fishcatch      854 vinnie      860 mu
no       886 chscase_geyser
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table] [Table]
[Table]