Paper: Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation (ICLR-2024)

Reviewer comment (Experiments):
The dataset used in this study is somewhat limited. Although there is confidence in the model's ability to generalize effectively to new datasets by representing data points in terms of their similarity to prototypes, there are concerns about its adaptability to other datasets. The issue of feature heterogeneity remains partly unresolved, as there may not be a latent state that aligns to all decision boundaries with the similarity measure. The top-K operation on prototypes helps to mitigate this problem to some extent but does not completely address it.

Author response:
**A1**: Thanks. We will respond from three aspects.

First, the feature heterogeneity is the main difficulty in training a model over multiple tabular datasets. Some existing methods deal with the problem by padding the feature, learning dimension-agnostic transformations, and using large language models. The uniform representation in this paper only encodes the common states of different models, while may lose the specific information. 

Second, for the given example, if we replace a feature 'x' with '1/x' (such transformation is reversible), the semantic meaning of this feature is changed. While in TabPTM, the mutual information between the feature and the label remains stable, and the feature will be emphasized in calculating the meta representation. 

Third, it is the relationship between instances, i.e., the distance between features, that determines the values in meta-representation. If two samples are close to each other under positive feature x, it will still be close when we reverse the feature. The meta-representation is robust to part of the transformations in practice.

Reviewer comment (Experiments):
Transformations applied to the features within the tables, such as replacing a feature 'x' with '1/x' (a reversible transformation where information is not altered or lost), might cause the hidden states to fail. This raises suspicion that the model might only exhibit robustness on certain datasets. It would be beneficial for the authors to clarify when the model is effective and when it is not; otherwise, this paper may potentially lead to misinterpretations in subsequent research.

Author response:
**A1**: Thanks. We will respond from three aspects.

First, the feature heterogeneity is the main difficulty in training a model over multiple tabular datasets. Some existing methods deal with the problem by padding the feature, learning dimension-agnostic transformations, and using large language models. The uniform representation in this paper only encodes the common states of different models, while may lose the specific information. 

Second, for the given example, if we replace a feature 'x' with '1/x' (such transformation is reversible), the semantic meaning of this feature is changed. While in TabPTM, the mutual information between the feature and the label remains stable, and the feature will be emphasized in calculating the meta representation. 

Third, it is the relationship between instances, i.e., the distance between features, that determines the values in meta-representation. If two samples are close to each other under positive feature x, it will still be close when we reverse the feature. The meta-representation is robust to part of the transformations in practice.

Reviewer comment (Experiments):
It might be advisable to consider using benchmark datasets like those in Taptap (https://arxiv.org/pdf/2305.09696.pdf) or Grinsztajn's work (https://proceedings.neurips.cc/paper_files/paper/2022/file/0378c7692da36807bdec87ab043cdadc-Supplemental-Datasets_and_Benchmarks.pdf) to enhance the credibility of the findings. Incorporating the average rank and rank standard deviation (like XTab) could provide deeper insights into this research.

Author response:
**A2**: Thanks for the suggestions. In addition to the classification results in the paper, we validate the robustness of the model in regression tasks, based on the benchmark datasets in the mentioned paper. 

In a regression task, the label $y$ of an instance $\boldsymbol{x}_i$ is a continuous value, so that we modify the meta representation of an instance $\phi(\boldsymbol{x}_i)$ as the concatenation of its distance with neighbours as well as their corresponding labels. In detail, if the $K$ nearest neighbours (based on the Euclidean distance) of instance $\boldsymbol{x}_i$ is ${\boldsymbol{x}_1, \dots, \boldsymbol{x}_K}$, we define the meta representation for regression as:

$\phi(\boldsymbol{x}_i) = [dist(\boldsymbol{x}_i, \boldsymbol{x}_1), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_j), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_K), y_1, \ldots, y_j, \ldots, y_K] \in \mathbb{R}^{2K}$.

Then, we can use a top-layer regressor to predict the label, i.e., $\hat{y_i}=\mathbf{T}_\Theta(\phi(\boldsymbol{x}_i))$. 

If $\mathbf{T}_\Theta$ is a linear model, the prediction could be implemented based on a neighbourhood manner by weighted averaging the labels of the neighbours.

We implement $\mathbf{T}_\Theta$ as a MLP. Similar to the protocol of classification we used in our paper, we collected 12 regression datasets. 9 of 12 datasets are sourced from Grinsztajn's benchmarks. We pre-train our TabPTM on half of them (house_sales, california_housing, MiamiHousing2016, ailerons, abalone, pole), and evaluate the learned model on the remaining datasets (analcatdata_supreme, wine_quality, house_16H, sulfur, yprop_4_1, houses). 

We set $K = 8$. By utilizing both Euclidean and Manhattan distance, the dimension of meta representation is $4K$. XGBoost and MLP are tuned on a single dataset. TabPTM is tuned on 6 pre-training datasets and generalizes to 6 unseen datasets without fine-tuning. The experiments are listed in the table below:

|                     | XGBoost | MLP    | TabPTM |
| ------------------- | ------- | ------ | ------ |
| analcatdata_supreme | 0.0849  | 0.0761 | 0.0838 |
| wine_quality        | 0.6691  | 0.4972 | 0.5787 |
| house_16H           | 0.6576  | 0.5564 | 0.4541 |
| sulfur              | 0.0234  | 0.0235 | 0.0199 |
| yprop_4_1           | 0.0301  | 0.0213 | 0.0207 |
| houses              | 0.2359  | 0.2169 | 0.2312 |

We will add more datasets in the future, and analyze the robustness of TabPTM.

Reviewer comment (Reproducibility):
There exist multiple versions of DANet, each differing in the number of layers (e.g., DANet-8, DANet-27). The layer configuration used in this paper should be specified.

Author response:
**A5**: Thanks. We use the default number of layers in the official code, i.e., DANet-20.

Reviewer comment (Novelty):
The "related work section" does not cover the huge literature on k-nearest neighbors and their multiple metric-based variants. It would require at least half a page to position this work amongst these methods.

Author response:
**A1**: Since the target of this paper is to learn a joint model over multiple heterogeneous tabular datasets, we do not emphasize metric-based methods a lot since the kNN or other metric-based methods cannot be applied across datasets. We discuss the relationship between ours and other meta-feature methods in "remarks" in Section 4.1. We will add more discussions in the related work part in the final version.

Reviewer comment (Theory):
The authors claim that their algorithm is "training-free". This may appear as a philosophical question, but to my opinion selecting class-representative items in a dataset is a -- strong and costly -- form of training, just as selecting the support vectors for a SVM is a form a training.

Author response:
**A2**: Thanks. In the paper, we first describe the idea of learning a general model with meta-representation, and the meta-representation aims to encode the relationship between an instance and its nearest prototypes in a particular class. We have tried various strategies, and in our experiments, we set *the whole training set* of a class as the prototype set. In other words, all training instances in a class are treated as prototypes. Therefore, the computational burden mainly lies in the similarity search, which could be easily accelerated in practical implementations. We show the training time comparison between various models, including the non-deep method and deep methods, in Table 2. The results indicate the training-free method makes predictions on a target dataset fast.

Reviewer comment (Novelty):
To my opinion this proposal is merely shifting the "not-that-hard" classification problem to the "not-that-easy" metric-learning and representative-choice problems.

Author response:
**A2**: Thanks. In the paper, we first describe the idea of learning a general model with meta-representation, and the meta-representation aims to encode the relationship between an instance and its nearest prototypes in a particular class. We have tried various strategies, and in our experiments, we set *the whole training set* of a class as the prototype set. In other words, all training instances in a class are treated as prototypes. Therefore, the computational burden mainly lies in the similarity search, which could be easily accelerated in practical implementations. We show the training time comparison between various models, including the non-deep method and deep methods, in Table 2. The results indicate the training-free method makes predictions on a target dataset fast.

Reviewer comment (Theory):
The main idea of the approach is a distance-based measurement between data records. However, it may seem somewhat trivial because the Lp norm-based metric (it appears that the authors used L1 or L2 norm) forms an Lp space, and this may be insufficient to represent tabular data, which exhibits complex distributions.

Author response:
**A1**: Thanks. Since diverse tabular datasets have different sets of features and classes, we investigate how to represent all the tabular datasets in a *uniform* form to support the training of a shared model. In our meta-representation, distances between one instance and its same-class nearest neighbours are utilized. We show the general idea of meta-representation and its basic form by using the L1 and L2 distance. Our experiments indicate that learning with a shared model can make up for information loss using the Lp norm-based metric. Furthermore, we can also concatenate other kinds of distance in meta-representation to improve its representation ability.

Reviewer comment (Theory):
Two heterogeneous tables can have different label distributions, even if feature columns have similar distributions between the two tables. Due to this inconsistent characteristic, fitting one table's distribution to that of another table may not work well.

Author response:
**A2**: That's right. The main challenge of learning a shared model over multiple datasets lies in the fact they have heterogeneous feature and label distributions. 

The main idea of using meta-representation to learn a general model across datasets is to capture the *intrinsic classification characteristic* over datasets. The meta-representation encodes the similarity distribution of a class. If an instance belongs to a class with high probability (e.g., near the center of a class or in the high-density area of the class), most distance values to its nearest neighbours are small. In contrast, the distance value between an instance and the non-target classes should have larger values. The top-layer module $T_\Theta$ captures such a kind of similarity distribution for classification, which could be shared across datasets. We also illustrate this idea in Figure 1 in the paper, and a discussion of what could be shared across datasets in Section 4.2.

Reviewer comment (Experiments):
The experimental results are insufficient to demonstrate the superiority of the proposed method.

Author response:
**A4**: Firstly, the baseline methods we compare are tuned specifically on individual datasets, making them more prone to achieving good performance, with computational costs increasing linearly as the dataset size grows. However, TabPTM is a model applied directly to all downstream tasks without the need for additional fine-tuning. Secondly, our TabPTM achieves the best performance in 3 of the 10 datasets in Table 1, and the two variants obtain the best two average results over the 10 downstream datasets. In Table 2, TabPTM achieves promising results with very short training/prediction time (300 times faster than XGBoost). We also validate the effectiveness of TabPTM on regression tasks (as shown on Respond to Reviewer XUXC A1).

Reviewer comment (Experiments):
As I understand, all the classes in all datasets are learnt as they are each individual class. Then how do you classify unseen data where for which classes are also unknown?

Author response:
**A5**: In our experiments, we evaluate our TabPTM on datasets with *unseen* classes and features, and the results validate the ability of TabPTM over datasets with unseen classes.

Given a new dataset, we calculate the meta-representation of an instance w.r.t. each one of the unseen classes. For example, given $C$ unseen classes, we obtain $C$ meta-representations with $K$-dimension. Then we apply the learned model $T_\Theta$ over them to obtain the $C$-dimensional class confidence vector without additional training. The instance is classified as one of the unseen classes by selecting the index with the maximum confidence.

Reviewer comment (Experiments):
Regression tasks which are plentiful in practice in tabular problems are not supported by the method.

Author response:
**A1**: Thanks for the suggestion. The meta representation of an instance encodes its distance with same-class nearest neighbours, and the idea could be extended to the regression scenario.
In a regression task, the label $y$ of an instance $\boldsymbol{x}_i$ is a continuous value, so that we modify the meta representation of an instance $\phi(\boldsymbol{x}_i)$ as the concatenation of its distance with neighbours as well as their corresponding labels. In detail, if the $K$ nearest neighbours (based on the Euclidean distance) of instance $\boldsymbol{x}_i$ is ${\boldsymbol{x}_1, \dots, \boldsymbol{x}_K}$, we define the meta representation for regression as:

$\phi(\boldsymbol{x}_i) = [dist(\boldsymbol{x}_i, \boldsymbol{x}_1), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_j), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_K), y_1, \ldots, y_j, \ldots, y_K] \in \mathbb{R}^{2K}$.

Then, we can use a top-layer regressor to predict the label, i.e., $\hat{y_i}=\mathbf{T}_\Theta(\phi(\boldsymbol{x}_i))$.

If $\mathbf{T}_\Theta$ is a linear model, the prediction could be implemented based on a neighbourhood manner by weighted averaging the labels of the neighbours. 

We implement $\mathbf{T}_\Theta$ as an MLP, and experiments validate its effectiveness to become a general regression model across datasets. Similar to the protocol of classification we used in our paper, we collect 12 regression datasets, pre-training our TabPTM on half of them (house_sales, california_housing, MiamiHousing2016, ailerons, abalone, pole), and evaluating the learned model on the remaining datasets (analcatdata_supreme, wine_quality, house_16H, sulfur, yprop_4_1, houses). It is notable that these two sets of datasets do not share the same set of features or regression ranges.

We set $K = 8$. By tilizing both Euclidean and Manhattan distance, the dimension of meta representation is $4K$. XGBoost and MLP are tuned on each dataset separately. TabPTM is tuned on 6 pre-training datasets and generalizes to 6 unseen datasets without fine-tuning. The experiments are listed in the table below:

|                     | XGBoost | MLP    | TabPTM |
| ------------------- | ------- | ------ | ------ |
| analcatdata_supreme | 0.0849  | 0.0761 | 0.0838 |
| wine_quality        | 0.6691  | 0.4972 | 0.5787 |
| house_16H           | 0.6576  | 0.5564 | 0.4541 |
| sulfur              | 0.0234  | 0.0235 | 0.0199 |
| yprop_4_1           | 0.0301  | 0.0213 | 0.0207 |
| houses              | 0.2359  | 0.2169 | 0.2312 |

TabPTM obtains the best performance on half of the datasets, and the second-best performance on the remaining datasets in a training-free manner.

Reviewer comment (Reproducibility):
Description of the tuning and evaluation protocols are not sufficient for reproducibility. Please provide a more detailed info on how you tuned and evaluated the baselines.

Author response:
**A2**: We described the implementation details, including the hyper-parameter tuning, in Section B.2. For some recently proposed methods such as TabCaps, we conducted hyperparameter tuning using their official code. We will provide more details in the final version. For example, hyper-parameters of XGBoost and MLP are selected by Optuna library with Bayesian optimization over 30 trials. Hyper-parameters of TabCaps are selected by hyperopt library over 30 trials. The hyperparameter spaces are as follows: 

- XGBoost: {'model': {'alpha': ['?loguniform', 0, 1e-08, 100.0], 'colsample_bylevel': ['uniform', 0.5, 1.0],'colsample_bytree': ['uniform', 0.5, 1.0], 'gamma': ['?loguniform', 0, 1e-08, 100.0], 'lambda': ['?loguniform', 0, 1e-08, 100.0], 'learning_rate': ['loguniform', 1e-05, 1], 'max_depth': ['int', 3, 10],'min_child_weight': ['loguniform', 1e-08, 100000.0], 'subsample': ['uniform', 0.5, 1.0]}}
- MLP: {"model": {"d_layers": ["$mlp_d_layers", 1, 5, 1, 512], "dropout": ["?uniform", 0.0, 0.0, 0.5 ], },"training": {"lr": ["loguniform", 1e-05, 2e-3],"weight_decay": ["?loguniform", 0.0, 1e-06, 1e-3]}}
- TabCaps: {'lr': uniform('lr', 4e-2, 2e-1), 'sub_class': randint('sub_class', 1, 5), 'init_dim': randint('init_dim', 32, 128), 'primary_capsule_size': randint('primary_capsule_size', 4, 32), 'digit_capsule_size': randint('digit_capsule_size', 4, 32), 'leaves': randint('leaves', 16, 64)}

Reviewer comment (Experiments):
TabPFNs do support datasets with around 10k samples (and TabPFN performance was shown to improve with more samples in the original paper). I believe TabPFN should be added to the comparison on larger datasets.

Author response:
**A4**: It is mentioned in the TabPFN paper that the method can only support small-scale tabular datasets. In our experiments, most large datasets contain over 10K samples. We do try TabPFN on larger datasets, while there exist computational issues on some datasets. Thanks for the suggestion. We also noticed that there are some improved variants of TabPFN recently, and we will compare some of them in the final version.

Reviewer comment (Experiments):
Number of datasets for which the model is evaluated is rather small by today's standards in tabular DL, especially given that the method generalizes without training. It would be great if you could also add results on the benchmark from `[1]`.

Author response:
**A5**: There are 22 classification datasets and 35 regression datasets in [1]. In our paper, we evaluate the results of TabPTM on 22 classification datasets with varying scale (some additional results are provided in Table 12 in Appendix d). Thanks for the suggestions. We evaluate TabPTM on 12 regression datasets (9 of the 12 datasets are sourced from benchmarks [1]), as described in Q1&A1. We will include additional datasets in the future.

Reviewer comment (Experiments):
What are the problem with supporting regression with a similar approach?

Author response:
**A1**: Thanks for the suggestion. The meta representation of an instance encodes its distance with same-class nearest neighbours, and the idea could be extended to the regression scenario.
In a regression task, the label $y$ of an instance $\boldsymbol{x}_i$ is a continuous value, so that we modify the meta representation of an instance $\phi(\boldsymbol{x}_i)$ as the concatenation of its distance with neighbours as well as their corresponding labels. In detail, if the $K$ nearest neighbours (based on the Euclidean distance) of instance $\boldsymbol{x}_i$ is ${\boldsymbol{x}_1, \dots, \boldsymbol{x}_K}$, we define the meta representation for regression as:

$\phi(\boldsymbol{x}_i) = [dist(\boldsymbol{x}_i, \boldsymbol{x}_1), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_j), \ldots, dist(\boldsymbol{x}_i, \boldsymbol{x}_K), y_1, \ldots, y_j, \ldots, y_K] \in \mathbb{R}^{2K}$.

Then, we can use a top-layer regressor to predict the label, i.e., $\hat{y_i}=\mathbf{T}_\Theta(\phi(\boldsymbol{x}_i))$.

If $\mathbf{T}_\Theta$ is a linear model, the prediction could be implemented based on a neighbourhood manner by weighted averaging the labels of the neighbours. 

We implement $\mathbf{T}_\Theta$ as an MLP, and experiments validate its effectiveness to become a general regression model across datasets. Similar to the protocol of classification we used in our paper, we collect 12 regression datasets, pre-training our TabPTM on half of them (house_sales, california_housing, MiamiHousing2016, ailerons, abalone, pole), and evaluating the learned model on the remaining datasets (analcatdata_supreme, wine_quality, house_16H, sulfur, yprop_4_1, houses). It is notable that these two sets of datasets do not share the same set of features or regression ranges.

We set $K = 8$. By tilizing both Euclidean and Manhattan distance, the dimension of meta representation is $4K$. XGBoost and MLP are tuned on each dataset separately. TabPTM is tuned on 6 pre-training datasets and generalizes to 6 unseen datasets without fine-tuning. The experiments are listed in the table below:

|                     | XGBoost | MLP    | TabPTM |
| ------------------- | ------- | ------ | ------ |
| analcatdata_supreme | 0.0849  | 0.0761 | 0.0838 |
| wine_quality        | 0.6691  | 0.4972 | 0.5787 |
| house_16H           | 0.6576  | 0.5564 | 0.4541 |
| sulfur              | 0.0234  | 0.0235 | 0.0199 |
| yprop_4_1           | 0.0301  | 0.0213 | 0.0207 |
| houses              | 0.2359  | 0.2169 | 0.2312 |

TabPTM obtains the best performance on half of the datasets, and the second-best performance on the remaining datasets in a training-free manner.

Reviewer comment (Reproducibility):
Could you describe the tuning and evaluation protocols for the baselines in more detail?

Author response:
**A2**: We described the implementation details, including the hyper-parameter tuning, in Section B.2. For some recently proposed methods such as TabCaps, we conducted hyperparameter tuning using their official code. We will provide more details in the final version. For example, hyper-parameters of XGBoost and MLP are selected by Optuna library with Bayesian optimization over 30 trials. Hyper-parameters of TabCaps are selected by hyperopt library over 30 trials. The hyperparameter spaces are as follows: 

- XGBoost: {'model': {'alpha': ['?loguniform', 0, 1e-08, 100.0], 'colsample_bylevel': ['uniform', 0.5, 1.0],'colsample_bytree': ['uniform', 0.5, 1.0], 'gamma': ['?loguniform', 0, 1e-08, 100.0], 'lambda': ['?loguniform', 0, 1e-08, 100.0], 'learning_rate': ['loguniform', 1e-05, 1], 'max_depth': ['int', 3, 10],'min_child_weight': ['loguniform', 1e-08, 100000.0], 'subsample': ['uniform', 0.5, 1.0]}}
- MLP: {"model": {"d_layers": ["$mlp_d_layers", 1, 5, 1, 512], "dropout": ["?uniform", 0.0, 0.0, 0.5 ], },"training": {"lr": ["loguniform", 1e-05, 2e-3],"weight_decay": ["?loguniform", 0.0, 1e-06, 1e-3]}}
- TabCaps: {'lr': uniform('lr', 4e-2, 2e-1), 'sub_class': randint('sub_class', 1, 5), 'init_dim': randint('init_dim', 32, 128), 'primary_capsule_size': randint('primary_capsule_size', 4, 32), 'digit_capsule_size': randint('digit_capsule_size', 4, 32), 'leaves': randint('leaves', 16, 64)}