Paper: TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications (ICLR-2024)

Reviewer comment (Experiments):
Regarding datasets, in my opinion, the "quality vs. quantity" balance should be improved. I believe that, for the field of tabular data, it is time to raise the bar in terms of dataset quality and to compose benchmarks that will have more chances to generalize to real world problems. After a quick review, I noticed the following datasets that can make the benchmark biased in ways that a hypothetical practitioner would not approve:  
- volcanoes-{a2,a3,a4,b1,b2,b5,b6,d1,d4,e1} -- the real world is not 10x biased towards tabular datasets about volcanoes with 3 features and 5 classes, but the benchmark is biased in this way.  
- wine-quality-{red,white} -- see the previous bullet.  
- fri_c0_1000_5, fri_c0_500_5 and 8 more similar tasks (10 in total) -- also seems to be a set of closely related problems as in previous bullets.  
- optdigits -- I think that computer vision problems should not be included in general tabular benchmarks, or should be presented in a separate group.  
- kr-vs-k -- I think that deterministic game-based problems (here, chess) should not be included in general tabular benchmarks, or should be presented in a separate group.  
- etc.  

Perhaps, works like `[1]` can be a source of more realistic datasets (worth mentioning: `[1]` is a bit limited in terms of dataset sizes, so other works like `[2]` may also be worth considering).

Author response:
```
(A) Regarding datasets, in my opinion, the "quality vs. quantity" balance should be improved.
```
We took the union of the collection from the AutoMLBenchmark (Gijsbers et al., 2022) and from the Auto-Sklearn 2 paper (Feurer et al., 2020), the exact rationale for dataset selection is currently detailed in Appendix C. 
Our motivation for extending those suites was that both works are prominent in the field of AutoML and in particular Gijsbers 2022 is the current most extensive empirical evaluation of AutoML methods.
We agree that the study [1] is also interesting but we believe the current collection covers sufficient variety (it contains the majority of datasets from [1] for instance).

Reviewer comment (Experiments):
I think that models should be more diverse. The current set of models is strongly biased towards:  
- tree-based models (all models except for MLP)  
- ensemble-like models (all models except for MLP)  

I am afraid that this may limit the potential of TabRepo in terms of what kind of analysis it allows conducting and what results it allows uncovering. I suggest considering the following:  
- Adding one linear model.  
- Adding one non-parametric model (e.g. kNN or modern kNN-like models), at least on datasets where it is possible.  
- Adding one modern parametric DL model (note that competitive parametric DL models are not necessarily heavy transformers `[3]`).  
- Adding one modern non-parametric DL model.  
- Keeping no more than two gradient boostings (personally, I would prefer just one, again, to reduce the bias, given that there is also RandomForest).  
- Excluding ExtraTrees.  

Also, I appreciate that there are various opinions on whether tabular DL models are worth attention. However, if DL models are not well presented, then the benchmark should be positioned as Classic-ML-only benchmark, but not as a general benchmark. Otherwise, some readers may have wrong expectations from the title and the abstract.

Author response:
```
(B) I think that models should be more diverse. The current set of models is strongly biased towards: tree-based models (all models except for MLP) ensemble-like models (all models except for MLP). I am afraid that this may limit the potential of TabRepo
```

We agree that more methods are worth adding, in particular DL methods, to increase model diversity and the usefulness of TabRepo.
Following your suggestion, we added: one linear model, one non-parametric model (KNN), one modern parametric DL model (FT-transformer), one modern non-parametric DL model (TabPFN). 
The results are described in the main response with a table. They follow the observations done in prior work [1,3], e.g. "modern" DL models are in between the performance of the best boosted trees methods and MLPs.

We do not think having only one gradient boosting method would be better for the dataset given that we have 3 DL methods now and boosted trees are performing better on average (the performance of the portfolio degrades significantly when removing CatBoost or LightGBM). We believe the methods are quite balanced between DL and trees with those additions.

Regarding excluding extra-trees, we also do not think it would make the dataset better. The method is useful in itself in particular given that it is extremely fast and can be useful in case one wants to find solution with very low fitting time (as can be seen in Fig 1).

Reviewer comment (Novelty):
In my opinion, the paper may need bigger stories (bigger than Section 4 and Section 5) to support the proposed dataset. My understanding is that it is (implicitly) suggested that TabRepo will help others to uncover and tell big/novel/non-trivial stories. However, compared to mainstream dataset-oriented works like `[1]` (where it is easy to imagine a wide target audience and a potential range of works based on the proposed benchmark), TabRepo seems to be more niche, and, to me, it is not immediately obvious how TabRepo can be used to obtain novel results. This is why, in this specific case, I expect the proposed dataset to be supported by at least one strong self-sufficient finding.  

I would like to add that:  
- I appreciate the stories told in Section 4 and Section 5, there is nothing wrong with them. However, they are not positioned as founding elements of the paper and, indeed, it may be too early to position them as such.  
- If Points 1 and 2 are perfectly addressed, then this point will not be a blocker, at least not for me.

Author response:
```
(C) My understanding is that it is (implicitly) suggested that TabRepo will help others to uncover and tell big/novel/non-trivial stories. However, compared to mainstream dataset-oriented works like [1] (where it is easy to imagine a wide target audience and a potential range of works based on the proposed benchmark), TabRepo seems to be more niche, and, to me, it is not immediately obvious how TabRepo can be used to obtain novel results.
``` 
We believe we demonstrated at least one impactful use-case for TabRepo which is reaching or outperforming the state-of-the-art of tabular predictions by simply considering ensemble of portfolio configurations. We agree that [1] provides valuable insight on the performance of DL versus tree-based methods. However, it serves a different purpose: it does not compare with SOTA tabular prediction methods (such as AutoGluon or AutoSklearn2 which use ensembling between different model families) and the dataset cannot be used to reach performance close to SOTA without spending significant compute given that all model would have to be retrained to evaluate the performance of ensembles. We believe our paper is serving more than a niche given that reaching SOTA performance at low cost is very important for research.

Reviewer comment (Writing):
Not a big issue, but personally, I find the "at no cost" wording a bit controversial. I would probably avoid it or somehow make it softer. The reported findings have non-zero cost, and the ability to use a public dataset at no cost is a usual property of public datasets. Perhaps, I am missing something here, but sharing this impression just in case.

Author response:
```
(D2) "I find the "at no cost" wording a bit controversial".
```
We agree with your point that "at no cost" is potentially misleading and we reworded the sentence to "at marginal cost". We believe that "at marginal cost" is suited given that running the experiments of the paper takes 2 hours of compute with TabRepo instead of 27K hours if models are retrained from scratch.

Reviewer comment (Experiments):
It is unclear how the results would transfer to a more out-of-distribution datasets. Would portfolio transfer work as well as AutoML or hyperparameter tuning on datasets that differ from the datasets present in the benchmark in some aspects. Seeing performance on disregarded larger datasets (discussed in appendix) could shed more light on practical applicability of TabRepo.

Author response:
```
It is unclear how the results would transfer to a more out-of-distribution datasets.
```
Regarding the generalization to out-of-distribution datasets, we evaluate portfolios on one unseen dataset via leave-one-dataset-out cross-validation. Of-course, the unseen dataset is still drawn from the same "distribution" of OpenML datasets however this gives a diverse distribution (as can be seen in Table 4-5-6 in the appendix).

Reviewer comment (Experiments):
One limitation that should be discussed is the tradeoff between computational efficency and memory. TabRepo is a large dataset, this could introduce problems in practice and make AutoML systems preferable.

Author response:
```
One limitation that should be discussed is the tradeoff between computational efficency and memory. TabRepo is a large dataset, this could introduce problems in practice and make AutoML systems preferable.
```
Thank for pointing this, we agree the tradeoff between efficiency and memory is important and we added a discussion of this point in the paper.
As pointed out in the appendix TabRepo takes 107GB as such having to store it in memory would impose expensive hardware. We bypassed this issue by using a memmap datastructure which allows to load the model predictions on the fly from disk to memory and allows to run the experiments on commodity hardware (only ~20GB of memory are needed). We added a mention of this point in appendix A).

Note that loading TabRepo is only required to perform portfolio selection, once those are learned an AutoML system can just store the list of models and re-use them on unseen tasks.

Reviewer comment (Evaluation):
Are the portfolios (selected models+hyperparameters) interpretable? (In a rough sense: there is a large MLP, small GBDT, heavily regularized MLP, etc.), or the portfolios are mostly random and change for different subsets (of datasets)?

Author response:
```
Are the portfolios (selected models+hyperparameters) interpretable? (In a rough sense: there is a large MLP, small GBDT, heavily regularized MLP, etc.), or the portfolios are mostly random and change for different subsets (of datasets)?
```
Yes indeed, the portfolios are interpretable and consistent when computed for different hold-out test dataset.
This consistency is expected as each portfolio shares 198 datasets in common with one-another (due to leave-one-dataset-out). 

Every portfolio contains the exact same first two models from CatBoost and MLP families. The third model is always from the LightGBM model family although it is not always exactly the same configuration. This portfolio pick order is somehow expected as tree models and neural networks ensemble well due to their lower correlation (which can be seen in Fig 1) and thus a MLP configuration is picked despite being a weaker model individually than other options. The early models all have low learning rates and are above average in model size (ex: number of leaves, depth, layers), but otherwise have hyperparameters that avoid the extremes of the search spaces. 

Interestingly, we observe that all 6 model families are picked at least once within the first 12 models in the portfolios which indicates that the model diversity is beneficial.

Reviewer comment (Experiments):
Could TabRepo results be used for a new, potentially OOD datasets (for example larger tabular datasets than present in the benchmark). How does zero shot portfolio transfer compares to AutoML and hyperparameter tuning on OOD datasets?

Author response:
```
Could TabRepo results be used for a new, potentially OOD datasets (for example larger tabular datasets than present in the benchmark). How does zero shot portfolio transfer compares to AutoML and hyperparameter tuning on OOD datasets?
```
Yes, we initially ran our evaluations with all datasets including the larger ones but observed the same results. Consequently, we decided to filter them as one obtained the same conclusion at lower compute cost. With larger datasets, we observed the same behavior e.g. that portfolio were outperforming AutoML systems which were outperforming ensembles of models from single families.

That being said, we believe one could improve over simple portfolio approaches which are agnostic to the dataset at hand by engineering dataset features. We chose to not include this approach in order to propose a simple baseline as we thought having a simple method perform on-par or better than current SOTA systems is a more interesting finding.

Reviewer comment (Novelty):
It's unclear to me how the problem of expensive benchmarks is solved by the proposed repository; at best it can be a benchmark for ensemble strategies. Something similar has been done in "CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and Salvageable Failure" by Purucker, though with much more involved ensembling methods.

Author response:
We agree that the ensembling method proposed in Purucker is very interesting. However, we chose to report only the performance of simple known methods (Caruana ensemble on top of greedy portfolio) because the the main point of our paper is to introduce a dataset and not a new method. In particular, we show that those simple known methods outperform or matche the performance of all AutoML systems and methods which we believe illustrate some of the benefit of the proposed dataset. That being said, we believe the work is very relevant to provide further improvement and added a reference in our paper.

Reviewer comment (Novelty):
This paper only uses a simple greedy method, similar to what is used in Autosklearn, or "Mining Robust Default Configurations for Resource-constrained AutoML" (Flaml zero shot) or "Learning Multiple Defaults for Machine Learning Algorithms" or "Learning hyperparameter optimization initializations".

Author response:
We agree that the ensembling method proposed in Purucker is very interesting. However, we chose to report only the performance of simple known methods (Caruana ensemble on top of greedy portfolio) because the the main point of our paper is to introduce a dataset and not a new method. In particular, we show that those simple known methods outperform or matche the performance of all AutoML systems and methods which we believe illustrate some of the benefit of the proposed dataset. That being said, we believe the work is very relevant to provide further improvement and added a reference in our paper.

Reviewer comment (Reproducibility):
The main reason that OpenML does not store predictions or probabilities is that this would be very storage intensive and there is no funding for it. Most works that do portfolio building have computed all of these metrics, though they are usually not shared since the storage overhead seems daunting.

Author response:
The storage is ~107GB (mentioned in the appendix C and F3) which is tractable. As you pointed out, most work does not store predictions and does not allow to efficiently measure the performance of ensemble configurations, this is a key contribution of our paper and dataset.

Regarding the difference with OpenML, we agree that the first two figures could have been generated with current results of OpenML. However, the key difference of TabRepo is to also expose the model predictions. This is a critical aspect as it allows researchers to consider ensembles and consequently to simulate solutions that outperforms SOTA AutoML systems at little cost (just recomputing metrics after having retrieved existing predictions). 

We believe that the key findings of the paper that the performance of SOTA tabular methods can be matched with simple techniques (as you said using only simple greedy methods) is a valuable finding for the community. We also think that providing a way to reach SOTA at low compute cost can be highly valuable for future research.