Paper: Towards Foundation Models for Learning on Tabular Data (ICLR-2024)

Reviewer comment (Experiments):
The foundation model is leveraging prior knowledge in the form of feature names + values. What role do these play in generalization to new tasks? If features are anonymous or have values scaled to an unexpected range (common occurances in practice) how much does this hurt generalization?

Author response:
Thank you for raising this insightful issue.
1. Feature names and values indeed serve as bridges between tabular data and language. Without the semantic information provided by feature names and values, we cannot effectively convey the necessary information to LLMs, thus impeding their ability to leverage prior knowledge when transferring to new tasks. However, even in the absence of meaningful feature names, the statistical learning ability of LLMs enhanced by GTL can still be utilized in few-shot instruction following and fine-tuning scenarios.
2. In practice, we can adopt several methods to express values that fall into unexpected ranges, thus partially avoiding such occurrences. For instance, we can **add units to scale the numerical values to a normal range, or use scientific notation to express extremely large or small values**.

Notably, we found that specifying the correct feature unit is important.
For example, in public datasets collected from Kaggle, height and weight are often essential features for predicting targets such as BMI or certain diseases. However, the unit for those features can differ across datasets. In bmidataset [1], the unit for height is cm and for weight is kg. But in predict-diabetes-based-on-diagnostic-measures dataset [2], the unit for height is inches and for weight is pounds.
Specifying the correct unit for each dataset helps to avoid distribution mismatches and misunderstandings for LLMs. This principle applies to other features like currency units (from dollar to EUR) and time units (from seconds to years).

[1] https://www.kaggle.com/datasets/yasserh/bmidataset

[2] https://www.kaggle.com/datasets/houcembenmansour/predict-diabetes-based-on-diagnostic-measures

Reviewer comment (Experiments):
For tabular learning, the presence of (potentially many) noisy / useless features is quite common. Does this hurt generalization of this method, as the model might pay attention to noisy or useless features?

Author response:
We appreciate your thoughtful emphasis on the potential impact of noisy and irrelevant features. While these methods intuitively improve generalization, due to limited resources, we have not yet conducted quantitative experiments to measure the impact of these factors specifically. But we would like to share how we try to address this issue:

1. When constructing high-quality training data, we remove certain useless features such as ID numbers or hash values. This helps to reduce the amount of noise in the data.
2. During the training process, we randomly drop a small percentage of the features. This strategy prevents the model from focusing excessively on one or a few features, thus promoting more balanced attention distribution across all relevant features.

Reviewer comment (Experiments):
For table 2, I'm a bit concerned about the gap between logistic regression and xgboost on these tasks. It seems like these datasets are simple enough that logistic regression matches xgboost performance. In more complicated real world cases, this gap is often quite greater, as the ability of xgboost to fit complex patterns in the data is needed for good performance. So, I'm concerned the evaluation isn't reflective of many real world uses cases. To be fair, in certain domains, such as medical domains, I've seen results demonstrating logistic regression leads to sota performance, but in many other domains this is not the case, so am unsure how much I can expect the utility of this model to generalize.

Author response:
Thank you for raising your concern about the evaluation datasets. Our current evaluation followed by TabLLM may indeed be relatively simple, contributing to the observed performance similarities. We would like to expand our evaluation to include a wider variety of heldout datasets and tasks from collected tabular data corpus. This will allow us to provide a more comprehensive view of the performance characteristics and generalizability of our model.

Reviewer comment (Experiments):
It's a bit strange to fit the baselines (LR, lightgbm, and xgb) in table 2 with no (?) hyperparameter tuning -- I didn't see a description of this process here. E.g., even with only 64 data points, its often possible to get considerably better results by tuning on LOO or k-fold CV on the training set for instance and considering a couple different hyperparameter combinations. Without this, I'm a somewhat suspicious the true performance of these baslines is a point or two higher than currently presented.

Author response:
Thank you for your question regarding the tuning process for the baselines. In fact, we did conduct hyperparameter tuning for all baselines. As detailed in the "Task-specific finetuning" section in Appendix B.1, we followed the hyperparameter tuning process outlined in TabLLM. This process involves **performing hyperparameter tuning based on 4-fold cross-validation on k-shots**. We apologize for any confusion caused by the lack of clarity in our paper and appreciate your diligence in pointing out this issue.

Reviewer comment (Evaluation):
The central results in Table 1 are unclear, partly due to the massive multi-row table format and partly due to the hazy conclusions. For example, on the Bank data, the only method that ever achieves AUC of 85 is LLaMA-GTL, but it does this with no examples (#contexts = 0), suggesting that LLaMA-GTL is great for this task but in-context learning doesn't help it. On C. Hous. and Car datasets, LLaMA-GTL strictly underperforms baselines with 0 contexts. On Diabetes, Heart, Income, and Jungle datasets, LLaMA-GTL again shows no better performance than 0-context baselines. This tells a less-than-compelling story about when LLaMA-GTL is the right tool, as existing methods seem more general with no context and no training. The GTL component does provide LLaMA with a significant boost in almost all cases.

Author response:
Thank you for highlighting this observation.
1. We pointed out in our paper that **"Introducing in-context examples does not necessarily lead to performance improvements over zero-shot inference in new tabular tasks"** (In section 4.1 Zero-shot vs In-context), which is not ambiguous. We believe this conclusion provides valuable insights about the existing conflict between general knowledge and statistical knowledge derived from a few in-context examples. Finding a trade-off between these two types of knowledge in current LLMs is indeed challenging.

2. When utilizing LLaMA-GTL as a tool, we recommend conducting zero-shot prediction when the task only involves common knowledge. Conversely, for domain-specific tasks or those requiring statistical learning, providing some examples for in-context learning would be more appropriate. Furthermore, in practice, an ensemble approach could be adopted to combine zero-shot and in-context learning for the final prediction. We are open to further discussion and welcome any suggestions you may have.

Reviewer comment (Experiments):
The results presented in Table 1 are also a weakness because the claims made following Table 1 include: (i) in-context learning doesn't help; (ii) GTL helps in most zero-shot settings; (iii) GTL helps in few-shot in-context settings. Point (i) is made clearly by the table but is a limitation—foundation models tend to do better with in-context examples, so if TabFMs don't, this needs to be investigated further. Points (ii) and (iii) are questionable comparisons since the number of in-context examples is presented as a control variable, but GTL models have significantly more training. While this may be a strength of GTL models over base models, it is simply an example of fine-tuning base models for specific tasks and feels like a less than novel finding.

Author response:
1. We would like to reiterate our first conclusion, which is that the effectiveness of in-context learning is largely dependent on the specific data and task at hand. Although the effectiveness of zero-shot and in-context learning varies with different datasets, it is clear that both capabilities are indispensable.

2. With GTL, the zero-shot and in-context learning abilities of LLM are both enhanced. Specifically, pretraining on large-scale diverse data enhances LLM's common knowledge on tabular tasks, improving its transferability to unseen datasets. Learning from data with context samples enhances LLM's ability to gain statistical knowledge from a few examples. For instance, on the C.Hous dataset, there is no performance gain for GTL in zero-shot settings compared to LLaMA, but with contexts=4 and 8, there is a notable improvement.

3. It is possible that in-context learning could have more stable effects through the use of more powerful models or more sophisticated prompt designs. We are working towards this goal and welcome further discussion and exchange of ideas.

Reviewer comment (Reproducibility):
How to apply the proposed model to new tasks? From Section A.3 of the appendix, my impression is that if I, as a researcher or a practitioner, want to use the proposed method as a baseline, I need:  
- "use GPT-4"  
- do "manual corrections"  
- "specify the task background"  
The above list looks relatively demanding (compared to traditional models) and gives a lot of room for making performance significantly worse/better without any intentions of doing so. In that regard, TabLLM looks easier to use. I think this aspect deserves more discussion (and perhaps should be positioned as a limitation).

Author response:
1. In actual applications, **our model can accept free-form input**. The purpose of constructing this data is to enable the model to use high-quality data during training, bringing it as close as possible to natural language processing.

2. In fact, **TabLLM also requires a significant amount of manual correction during the serialization process**. For instance, each column name and feature value are manually mapped to values that are more similar to natural language. We leveraged GPT-4 to assist us in generating this information due to the richness and volume of our dataset.
We acknowledge that implementing our method may be more demanding compared to traditional models, but we believe that the additional effort required are necessary towards tabular foundation models.

Reviewer comment (Theory):
Conceptually important claims.  
Quote 1: “Besides, using a text representation enables the easy integration of crucial meta information that can hardly be utilized by traditional studies for learning on tabular data, such as feature meanings and background knowledge.” This suggests that traditional models’ inability to process textual information is a limitation worth addressing. However:  
- If a model requires task and feature descriptions, it is a limitation for applying it to non-annotated tasks, whereas traditional models are free from this limitation.  
- I am not aware of studies showing that average tabular problems contain significant useful information in their descriptions; it might be true for tiny tasks, but is less obvious for larger tasks.  
Quote 2: “it also showcases performance that approaches, and in instances, even transcends, the renowned yet mysterious closed-source LLMs like GPT-4.” I recommend providing a relevant citation to support positioning GPT-4 as a strong baseline for the considered scope, or else remove the emphasis on GPT-4 (though using it as a baseline is fine).

Author response:
1. Indeed, for tasks lacking semantic information, the general prior knowledge of LLMs may not be helpful. This can be viewed as a challenge rather than a limitation. Therefore, we need LLMs to have in-context learning and fine-tuning capabilities. These abilities require data knowledge and statistical learning, areas where current LLMs fall short. The comparison between LLaMA-GTL and LlaMA shows promising improvements brought by our method, and we hope this can inspire future research in this field.

2. Thank you for your suggestion. We recognize GPT-4 as a general and widely-used model that serves as a baseline in many domains. However, as GPT-4 is a closed-source, black-box model with undisclosed training data and technical details, it has an element of mystery to it. Through our work, we aim to improve upon existing open-source efforts, particularly in the field of tabular learning, to approach or even surpass the performance of such closed-source general models. Our intent is to help progress the research in this domain.

Reviewer comment (Experiments):
There is a possibility that modern well-trained LLMs like LLaMA memorized the datasets being used as the benchmark in this paper. Almost all datasets were present on the internet (some plentifully) before the knowledge cutoff for those models. Additional experiments on newer datasets or explicit discussion and testing for memorization are necessary to claim remarkable zero-shot performance on tabular tasks.

Author response:
Thank you for your insightful question. We agree with your assertion that there is a potential for LLMs to memorize public datasets. This hypothesis is indeed plausible, as corroborated by our experiments, as detailed in Table 1. In zero-shot experiments, we observed notably high AUROC values for various models across different datasets. For instance, GPT4 exhibited a strong performance on the Diabetes and Heart datasets, while T0 performed exceptionally well on the Car and Income datasets.

However, in the case of LLaMA, the performance across all nine evaluation datasets was below 0.7. More specifically, the scores for the Bank and Diabetes datasets were even lower, at 0.57 and 0.66 respectively. These results suggest that LLaMA has neither learned the distribution of these datasets nor memorized them.

Substantially, one of our major findings is that "LLaMA-GTL significantly improves LLaMA in most zero-shot scenarios," particularly on the aforementioned two datasets. This improvement underscores the knowledge transfer capability of generative learning, which allows LLMs to enhance their performance without data memorization.

Reviewer comment (Evaluation):
Looking at Table 1, the results vary significantly depending on the number of shots (for the same model: see GPT-4 on Diabetes for example), knowing that LLMs are very sensitive to prompting (even prompt formatting `[1]`). Some standard deviation over shot-selection or prompt formats is needed in this table to make conclusions.

Author response:
We appreciate your keen observation on the influence of **shot selection and prompt formatting** on model performance. We concur with your insight and would like to delve deeper into our findings in these areas.

On one hand, as highlighted in Section 4.1 of our paper, we observed that **"few in-context examples do not provide robust statistical knowledge",** which underscores the importance of shot selection. We have investigated several shot-selection strategies and noted that LLMs' predictions for a specific sample are sensitive to the chosen shots. To better align with real-world scenarios and enhance the stability of our experimental results, we have adopted a strategy of randomly selecting context samples for each test sample.

On the other hand, our findings indicate that prompt formatting can also significantly influence the performance of LLaMA, T0, and GPT. For instance, in zero-shot setting, through very few feature prompts modifications, performance of TabLLM will decrease a lot. **All of those LLMs without GTL demonstrated high sensitivity to prompt formatting in our evaluations**. The impact of prompt formatting is evident in both the serialization of a single sample and the additional instruction for context samples.  We aimed to explore the potential of each LLM by employing diverse versions of prompt formats, as depicted in Figure 4.

Reviewer comment (Experiments):
The baselines that propose foundation models for tabular data without LLMs are discussed in related work, but not compared against in the few-shot experiments, I believe this is an important comparison for a paper to claim a "comprehensive comparison".

Author response:
Our approach aims to deliver not only task-specific finetuning but also two additional capabilities that are equally crucial: transferring knowledge to new tasks for zero-shot predictions, and quickly adapting to data-specific knowledge with few-shot examples via in-context learning. While the mentioned baselines, XTab and UniTabE, also conducting pretraining on large-scale data, they are more focus on fine-tuning. Our method can simultaneously demonstrate all three capabilities.

More specifically:

We appreciate UniTabE [1] for their effort in collecting a 13B dataset and their unique architecture design, which strengthens the association between column names and values. In the finetuning scenario, it shows promising results over many baselines across seven public datasets. However, there are a few concerns:
1. **UniTabE did not make comparisons with other LLMs in the zero-shot scenario**, only comparing their method with random initialization and their own finetuned model, which does not provide meaningful insights.
2. In Table 4 of their paper, **UniTabE only displayed the accuracy metric in the zero-shot scenario, which differs from the AUC metric** used in Table 3. This might not be an effective way to evaluate model performance, especially for imbalanced datasets.
3. While UniTabE claims to utilize free-form prompts for tuning pretraining and finetuning, their **1-layer LSTM decoder lacks the language ability and prior knowledge compared to LLMs**. This might limit their ability to support free-form questions in downstream tasks without finetuning on the exact same task.

As for XTab [2], they utilize carefully designed featurizers and training objectives for cross-table pretraining, enhancing learning speed and performance. However:
1. Their data-specific design might not be applicable to zero-shot and in-context learning scenarios.
2. The improvement their method offers over the FT-Transformer baseline is not particularly significant **in heavy finetuning setting** with a win rate of less than 60% over all tasks.

In light of these points, we believe that our method offers distinct advantages. We appreciate your insightful question and will consider including such comparisons in our future work to provide a more comprehensive evaluation.

[1] Yang, Yazheng, et al. "UniTabE: Pretraining a Unified Tabular Encoder for Heterogeneous Tabular Data." arXiv preprint arXiv:2307.09249 (2023).

[2] Zhu, Bingzhao, et al. "XTab: Cross-table Pretraining for Tabular Transformers." arXiv preprint arXiv:2305.06090 (2023).