Paper: ExcelFormer: Making Neural Network Excel in Small Tabular Data Prediction (ICLR-2024)

Reviewer comment (Experiments):
(major) **Many *orthogonal* changes (listed in the summary above) are proposed *at once*.** It makes it difficuilt to attribute the observed results to any single element, which I believe to be important in the research context, especially for this genre of papers. I believe that the elements should be introduced either in isolation or step-by-step, but not at once (unfortunately, ablating each of the elements using the *final* architecture does not addresses the issue). Also, in my opinion, each of the elements should be compared against existing alternatives (i.e. the proposed augmentations VS existing augmentations, the proposed embeddings VS the existing embeddings, etc.). Overall, modifying the well-established Transformer architecture in six(!) different aspects (listed in the summary), most of which has dedicated research subfields looks like an extremely ambitious goal to me. And I respect that, however, it makes it extremely hard to properly introduce and analyse each of the elements.

Author response:
Thank you for your concerns. We would like to clarify it from 2 aspects:

**(1) More detailed studies**. We have included more ablation studies and addictive studies to inspect the performances of proposed components.

1. Addictive study: on a vanilla Transformer (Tra), we show that **all of our proposed modules**, SPA, IAI, and GLU projection/embedding (GLU projection and embedding share the idea, and both of them process features through the GLU) **make positive contributions to tabular data prediction tasks**, and the contributions are cumulative.

| | Tra | Tra + IAI | Tra + SPA | Tra + SPA + IAI |Tra + SPA + IAI + GLU|
|-|-|-|-|-|-|
| binclass (Ave Norm AUC, ↑) | 0.232±0.36 | 0.683±0.37| 0.289±0.37 | 0.753±0.33  | 0.775±0.30|
| regression (Ave Norm nRMSE, ↑) | 0.138±0.30  | 0.738±0.25  | 0.751±0.26|0.766±0.35 | 0.911±0.16|

2. Compare our data augmentation with Mixup and CutMix. Ablation study in Figure 4 show CutMix and Mixup perform worse than our proposed data augmentation. To analyze the difference of cutmix and Feat-Mix carefully, in Table 6, we present the results of cutmix and Feat-Mix on several datasets. Theoretically, the difference between cutmix and feat-mix lies in the use of feature importance. Empirically, we find that:

* In general, **Feat-Mix outperforms Cutmix** due to its ability to provide more accurate labels by considering feature importance.

* When noise exists in the data, **Feat-Mix is less affected by the noise while cutmix is more heavily impacted**. Since tabular datasets have been shown to contain many uninformative features`[1]`, we believe Feat-Mix is a superior choice.

The detailed results:

| | Breast | Diabetes | Campus | cpu | fruitfly | yacht |  
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| CutMix (↑) | 0.702 | 0.822 | 0.972 | -102.06 | -16.19 | -3.59 |   
| Feat-Mix (↑) | **0.713** | **0.837** | **0.980** | **-79.10** | **-15.86** | **-0.83** |   
| CutMix (on noisy data) (↑) | 0.688 | 0.809 | 0.938 | -115.10 | -17.09 | -4.40 |
| Feat-Mix (on noisy data) (↑) | **0.700** | **0.834** | **0.969** | **-74.56** | **-16.60** | **-0.89** |  
| $\Delta$ CutMix (↓) | 0.014 | 0.013 | 0.034 | 13.04 | 0.90 | 0.81 |   
| $\Delta$ Feat-Mix (↓) | **0.013** | **0.003** | **0.011** | **-4.54** | **0.74** | **0.06** |

3. Compare Hid-Mix, Feat-Mix with Feature resample (an augmentation for tabular data) on MLP: 

The results are presented below. We observe that Hid-Mix outperforms Feature Resample and all these approach dominate different parts of datasets.

|   | Feature Resample | Feat-Mix | Hid-Mix | No data augmentation |
|-|--|--|-|--|
| Binclass (Ave Norm AUC, ↑) | 0.576±0.42 | 0.530±0.41 | 0.614±0.41 | 0.436±0.42 |
| regression (Ave Norm nRMSE, ↑)|0.661±0.39|0.629±0.40 | 0.681±0.39 | 0.334±0.42 |

4. An ablation study is shown in Figure 4 that the performances of Excelformer drops when removing one of those components.

**(2) Why propose those components?**

1. We would like to point out a specific issue of the tabular data prediction domain: **Tabular datasets are highly diverse and heterogeneous. Unlike other fields (such as computer vision) where a single model can dominate most datasets, different previous tabular data prediction models only excel on a part of datasets (see Table 2: excluding Excelformer, the best models vary in different groups). Excelformer is the first work that realizes such dominance.**

2. Given the diversity of datasets in this field and the no-free-lunch principle, finding a single component that is universally applicable to all types of datasets is challenging (previous works have faltered in this aspect). **In fact, the orthogonality of methods is indeed the cornerstone of our contribution. When addressing the diversity of tabular datasets and tasks, the proposal and integration of orthogonal methods augment the robustness and efficacy of Excelformer**. Then, we achieve:

*  **Using default hyperparameters (it is very convenient in application!)**, ExcelFormer outperforms previous hyperparameter-tuned models;

* ExcelFormer achieves robust performance **across various types of datasets / tasks** (see Table 2)

**We argue that AI development aims to facilitate practical applications.** The combination of versatility and the convenience of not requiring hyperparameter tuning positions Excelformer as an excellent practical tool for tabular data prediction tasks, especially for users with limited computer science proficiency or computational resources. **Please note that no previous work attains performances as dominating as Excelformer, making this a key contribution.**

---

References:

`[1]` Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS, 2022.

Reviewer comment (Experiments):
(major) In my opinion, **the storyline around rotation invariance should be extended with specific analysis/experiments/results. Purely intuitive guidance may not be enough to drive the design decisions.** There are multiple places where the *formal* term "rotation invariance" is used in *informal* ways. For example, the paper uses terms like "more/nearly non-rotationally invariant". Overall, there is nothing wrong with relying on intuition, but after a certain threshold, there is a risk of coming to wrong conclusions. A potential solution is to design a dedicated experiment that will quantify rotation invariance of any ML model. Then, some of the proposed elements can be motivated as a way to reduce the invariance according to the designed experiment. Again, this should be done *when introducing the elements*, not with the final architecture (as in Figure 5).

Author response:
Thank you for your constructive suggestion! In the initial version, we conducted an analysis of the rotation invariance property, i.e. the subfig (a) in Figure 5. Following your suggestion, we have taken a step further by providing an additive study and an ablation study to analyze the effects of SPA and IAI on breaking the rotation invariance of deep learning models. 

* Additive study setting: We use SPA and IAI on the backbone of FTT `[1]`

* Ablation study setting: We exclude SPA and IAI from the architecture of Excelformer

We find that:

* With SPA and IAI, our Excelformer shows better non-rotationally invariance, compared with previous works.

* On the FTT backbone, the addition of SPA and IAI makes positive contributions to break the rotational invariance property.

* On the Excelformer architecture, removing SPA and IAI weakens Excelformer's rotational invariance property.

* **The effects of adding or removing SPA and IAI are cumulative.**

Reviewer comment (Novelty):
Unfortunately, in my opinion, **the novelty is limited.** Some of the proposed modifications (listed in the summary above) are technically new, however, from the same technical perspective, they remain similar to the existing alternatives.

Author response:
Thank you for this concern. I believe the evaluation of novelty is quite objective. I suggest considering the changes an approach brings as a reference for novelty creation. Let's examine what changes we made by introducing Excelformer:

* BEFORE EXCELFORMER:

  * The overall performance of deep learning approaches is **inferior** to GBDTs.
  * Different previous models excels in specific types of datasets / tasks (see Table 2).  It is **challenging to determine a "good" model** before conducting an evaluation.
  * **Fine-tuning hyperparameters is crucial** for attaining state-of-the-art results across all previous models. This poses challenges for users with limited proficiency in computer science or access to computational resources.

* AFTER EXCELFORMER IS PROPOSED:
  * Excelformer (a DL approach) beats GBDTs and various DL competitors;
  * Excelformer outperforms previous models on various types of datasets / tasks (see Table 2). It is easier for users to choose a model (Excelformer) now.
   * Although fine-tuning hyperparameters is beneficial for Excelformer, using its default hyperparameters can achieve top results.

* To achieve this, THE COST WE INCUR IS MINIMAL:
  * Obviously, the computational efficiency of SPA is the same as vanilla self-attention module.
  * Interaction Attenuated Initialization is almost computational free.
  * As analyzed in Sec. 2.2, the computational complexity of GLU projection is equivalent to that of vanilla FFN.
  * Feat-Mix and Hid-Mix are nearly computational free.

We will highlight the contribution of Excelformer in the final version.

Considering the changes that Excelformer has brought to the field of tabular data prediction, we would appreciate it if you could reconsider your evaluation of novelty. Thank you!

Reviewer comment (Reproducibility):
In my opinion, sharing code, starting from the review stage, is important for this kind of studies. I wish I had an opportunity to have a look at the code to review the experimental setup and implementation details.

Author response:
We would like to make all codes publicly available, including detailed implementations, model training and experiment examples, along with user-friendly APIs. Given the substantial volume of implementation and experiment codes, we chose not to include them in the initial submission to allow more time for the improved organization and presentation of this project. **The code is now included in the supplementary materials**, and we will continue to refine the project to offer more comprehensive documentation and unified APIs.

Reviewer comment (Writing):
I recommend proof-reading the paper for English style, vocabulary and grammar issues.

Author response:
Thank you for your careful suggestions. We will do careful proof-reading for the final version.

Reviewer comment (Reproducibility):
How exactly is the mutual information computed for the continuous features and for regression labels?

Author response:
In the implementation, we utilize the scikit-learn package to calculate mutual information. Specifically, for classification tasks, we employ the "feature_selection.mutual_info_classif" function, and for regression tasks, we use the "feature_selection.mutual_info_regression" function. Thank you for your reminder! We have depitcted the implementation in Appendix F.

Reviewer comment (Writing):
The paper can be written better, typos exist here and there throughout the manuscript. (I will list a few of them in the questions section)

Author response:
**A2**: Thank you for your careful suggestions. We have corrected those typos in the revised version and will do careful proofreading for the final version.

Reviewer comment (Experiments):
In table 1, an interesting investigation would be how ExcelFormer would behave without any data augmentation (compared to the rest, not the ablation that is given) or how FTT would perform with the proposed augmentation approaches. I am additionally surprised that CatBoost performs worse compared to XGBoost consistently.

Author response:
**A1**: Thank you for your thoughtful considerations. We would like to address the aforementioned questions together as they are closely related.

(i) Certainly, we intend to make all codes publicly available, including detailed implementations, model training and experiment examples, along with some user-friendly APIs. We are confident that Excelformer can serve as a convenient and versatile solution for tabular data prediction. Given the substantial volume of implementation and experiment codes, we chose not to include them in the initial submission to allow more time for the improved organization and presentation of this project. **The code is now included in the supplementary materials**, and we will continue to refine the project to offer more comprehensive documentation and unified APIs.

(ii) **Why previous SOTA be debunked?** Without intending any offense, we would like to explain why you find that recent approaches (e.g., TabNet, Node, SAINT) initially claimed state-of-the-art performance but were later "debunked". Actually, tabular datasets are highly diverse and heterogeneous. Unlike other fields (such as computer vision) where a single model can dominate most datasets, **different previous tabular data prediction models only excel on a part of datasets** (see Table 2: excluding Excelformer, the best models vary in different groups). In this paper, Excelformer tries to realize such dominance. Previous works ignored such dataset diversity, and most of them (e.g., TabNet, Node, SAINT) only validated their claims on limited datasets. Similarly, the conclusions in the paper titled *Tabular data: Deep learning is not all you need* may also not be entirely convincing, as it relied on only 11 datasets. Hence, in our paper, **we extensively utilized 130+ diverse datasets, and also presented results categorized by characteristics in Table 2**. See Table 2, one may observe that different previous methods exhibit preferences for distinct datasets, but Excelformer obtains top performances in all the groups.

(iii) **Why propose several components?** Given the diversity of datasets in this field and the no-free-lunch principle, it is challenging to find a component widely applicable to all the dataset types (under default hyper-parameter setting). Following your suggestion, we compare the Excelformer (without data augmentation) against previous models, and found Excelformer (without data augmentation) achieves the best performance (see `Q4 & A4`). However, as an application-oriented research, we prioritize both model performance and user convenience. Thus, in this paper, we introduce those useful components, and we find they can make Excelformer:

* exempt from hyper-parameter tuning (using default hyperparameters is very convenient in application!)

* capable of robust performance across various types of datasets and tasks (see Table 2)

We believe that the purpose of AI research is to facilitate practical applications. The combination of versatility and the convenience of not requiring hyperparameter tuning positions Excelformer as an excellent practical tool on tabular data prediction tasks, especially for users with limited computer science proficiency or computational resources.

(iv) **Why XGBoost outperforms CatBoost?**

* Firstly, in our experiments, **XGBoost does not consistently outperform CatBoost**. See Table 3, under default parameters, CatBoost outperforms XGBoost. Additionally, see Table 2, CatBoost performs better than XGBoost in classification and the case of #. Features $\ge$ 16. Besides, CatBoost typically outperforms XGboost in multi-class classification (see Table 7 of the revised version). **Such distinction is due to the diversity of datasets and tasks.**

* A key contribution of CatBoost is reducing the workload for users in terms of feature engineering, visualization, and hyper-parameter tuning. This enables CatBoost to outperform XGBoost when using default parameters. However, after conducting extensive hyperparameter search for GBDTs (see Tables 6 and 7 - we provide more estimator candidates (4096) than previous works), we find that XGboost can outperform CatBoost on over 50% of datasets.

**In summary, Excelformer is an application-oriented research. It achieves robust performance across diverse datasets using default hyperparameters. The proposed components work together and make Excelformer superior on the diversity of tabular data with almost no added cost.**

Reviewer comment (Experiments):
No multi-class classification problems in the 96 datasets for the small-scale tabular datasets and only 4 datasets in the 21 large-scale datasets. 4 datasets in 117 datasets is an underrepresentation.

Author response:
Thank you for your constructive suggestion. We use fewer multi-class classification datasets because multi-class classification tasks themselves occur relatively infrequently among tabular data prediction tasks, especially on smaller-scale datasets. Some previous works have not included multi-class classification datasets`[1]`.

Following your suggestion, we provide the results on 16 **additional multi-class classification datasets** (refer to Table 7 in the revised version), and we find that **Excelformer significantly performs best among various representative approaches**. We will included these results in the final version.

| | XGb (t)       | Cat (t)  | FTT(t)  | MLP(t) | DCNv2(t)   | AutoInt(t)  | SAINT(t) | Excelformer(t)  |
|---|----|-|--|-|-|-|--|--|
| Rank($\pm$std) {$\downarrow$}    | 3.91$\pm$2.28     | 2.53$\pm$1.70     | 5.53$\pm$1.85    | 6.09$\pm$1.88    | 5.97$\pm$1.61    | 4.66$\pm$1.65    | 5.22$\pm$1.41    | **2.09$\pm$1.23** |

Reviewer comment (Evaluation):
Regarding the evaluation metrics, why would the authors use AUC for binary classification and ACC for multi-class classification? The latter would not be a good metric for imbalanced datasets.

Author response:
**A6**: ACC is the standard metric for multi-class classification tasks, which is widely used in previous tabular data prediction papers `[2,3,4]`. We try to follow these works so that offer easy reference for followers.

Reviewer comment (Reproducibility):
Without code release, I find it difficult to trust the results, as unfortunately there exist a plethora of recent DL architectures that claim state-of-the-art performance (TabNet, Node, Saint, etc) only to be debunked later on. It is necessary to validate the proper setup of the baseline algorithms and to verify the results of the method.

Author response:
**A1**: Thank you for your thoughtful considerations. We would like to address the aforementioned questions together as they are closely related.

(i) Certainly, we intend to make all codes publicly available, including detailed implementations, model training and experiment examples, along with some user-friendly APIs. We are confident that Excelformer can serve as a convenient and versatile solution for tabular data prediction. Given the substantial volume of implementation and experiment codes, we chose not to include them in the initial submission to allow more time for the improved organization and presentation of this project. **The code is now included in the supplementary materials**, and we will continue to refine the project to offer more comprehensive documentation and unified APIs.

(ii) **Why previous SOTA be debunked?** Without intending any offense, we would like to explain why you find that recent approaches (e.g., TabNet, Node, SAINT) initially claimed state-of-the-art performance but were later "debunked". Actually, tabular datasets are highly diverse and heterogeneous. Unlike other fields (such as computer vision) where a single model can dominate most datasets, **different previous tabular data prediction models only excel on a part of datasets** (see Table 2: excluding Excelformer, the best models vary in different groups). In this paper, Excelformer tries to realize such dominance. Previous works ignored such dataset diversity, and most of them (e.g., TabNet, Node, SAINT) only validated their claims on limited datasets. Similarly, the conclusions in the paper titled *Tabular data: Deep learning is not all you need* may also not be entirely convincing, as it relied on only 11 datasets. Hence, in our paper, **we extensively utilized 130+ diverse datasets, and also presented results categorized by characteristics in Table 2**. See Table 2, one may observe that different previous methods exhibit preferences for distinct datasets, but Excelformer obtains top performances in all the groups.

(iii) **Why propose several components?** Given the diversity of datasets in this field and the no-free-lunch principle, it is challenging to find a component widely applicable to all the dataset types (under default hyper-parameter setting). Following your suggestion, we compare the Excelformer (without data augmentation) against previous models, and found Excelformer (without data augmentation) achieves the best performance (see `Q4 & A4`). However, as an application-oriented research, we prioritize both model performance and user convenience. Thus, in this paper, we introduce those useful components, and we find they can make Excelformer:

* exempt from hyper-parameter tuning (using default hyperparameters is very convenient in application!)

* capable of robust performance across various types of datasets and tasks (see Table 2)

We believe that the purpose of AI research is to facilitate practical applications. The combination of versatility and the convenience of not requiring hyperparameter tuning positions Excelformer as an excellent practical tool on tabular data prediction tasks, especially for users with limited computer science proficiency or computational resources.

(iv) **Why XGBoost outperforms CatBoost?**

* Firstly, in our experiments, **XGBoost does not consistently outperform CatBoost**. See Table 3, under default parameters, CatBoost outperforms XGBoost. Additionally, see Table 2, CatBoost performs better than XGBoost in classification and the case of #. Features $\ge$ 16. Besides, CatBoost typically outperforms XGboost in multi-class classification (see Table 7 of the revised version). **Such distinction is due to the diversity of datasets and tasks.**

* A key contribution of CatBoost is reducing the workload for users in terms of feature engineering, visualization, and hyper-parameter tuning. This enables CatBoost to outperform XGBoost when using default parameters. However, after conducting extensive hyperparameter search for GBDTs (see Tables 6 and 7 - we provide more estimator candidates (4096) than previous works), we find that XGboost can outperform CatBoost on over 50% of datasets.

**In summary, Excelformer is an application-oriented research. It achieves robust performance across diverse datasets using default hyperparameters. The proposed components work together and make Excelformer superior on the diversity of tabular data with almost no added cost.**

Reviewer comment (Experiments):
Can you include more multi-class classification problems in the used benchmarks and provide results?

Author response:
Thank you for your constructive suggestion. We use fewer multi-class classification datasets because multi-class classification tasks themselves occur relatively infrequently among tabular data prediction tasks, especially on smaller-scale datasets. Some previous works have not included multi-class classification datasets`[1]`.

Following your suggestion, we provide the results on 16 **additional multi-class classification datasets** (refer to Table 7 in the revised version), and we find that **Excelformer significantly performs best among various representative approaches**. We will included these results in the final version.

| | XGb (t)       | Cat (t)  | FTT(t)  | MLP(t) | DCNv2(t)   | AutoInt(t)  | SAINT(t) | Excelformer(t)  |
|---|----|-|--|-|-|-|--|--|
| Rank($\pm$std) {$\downarrow$}    | 3.91$\pm$2.28     | 2.53$\pm$1.70     | 5.53$\pm$1.85    | 6.09$\pm$1.88    | 5.97$\pm$1.61    | 4.66$\pm$1.65    | 5.22$\pm$1.41    | **2.09$\pm$1.23** |

Reviewer comment (Experiments):
Is the EXCELFORMER more non-rotationally invariant and more noise insensitive? In my perspective, an interesting addition would be to include the plain architecture of ExcelFormer in the investigation, then with every suggestion included one at a time (SEA, IAI), then both. This would show how the architecture gets more non-rotationally invariant as the different components are added compared to the beginning. Comparing against FTT is interesting, but it does not separate the impact of the overall differences in the architecture vs (SEA, IAI).

Author response:
**A8**: Thank you for your constructive suggestion. We conduct an additive study and an ablation study (see Fig. 5 of the revised version) to inspect the benefits introduced by SPA and IAI on breaking the rotational invariance property of DL models. We find that:

* With SPA and IAI, our Excelformer shows better non-rotationally invariance, compared with previous works.

* On the FTT backbone, the addition of SPA and IAI makes positive contributions to break the rotational invariance property.

* On the Excelformer architecture, removing SPA and IAI weakens Excelformer's non-rotational invariance property.

* **The effects of adding or removing SPA and IAI are cumulative.**

Thank you once again for your valuable time and constructive feedback. We have taken your feedback into careful consideration and have made revisions to the manuscript accordingly. We kindly request you to reconsider the ranking, considering the improvements (especially the detailed ablation and additive studies)  made and the significant practical value that ExcelFormer offers.

Reviewer comment (Reproducibility):
I would urge the authors to provide the code to reproduce the results.

Author response:
**A1**: Thank you for your thoughtful considerations. We would like to address the aforementioned questions together as they are closely related.

(i) Certainly, we intend to make all codes publicly available, including detailed implementations, model training and experiment examples, along with some user-friendly APIs. We are confident that Excelformer can serve as a convenient and versatile solution for tabular data prediction. Given the substantial volume of implementation and experiment codes, we chose not to include them in the initial submission to allow more time for the improved organization and presentation of this project. **The code is now included in the supplementary materials**, and we will continue to refine the project to offer more comprehensive documentation and unified APIs.

(ii) **Why previous SOTA be debunked?** Without intending any offense, we would like to explain why you find that recent approaches (e.g., TabNet, Node, SAINT) initially claimed state-of-the-art performance but were later "debunked". Actually, tabular datasets are highly diverse and heterogeneous. Unlike other fields (such as computer vision) where a single model can dominate most datasets, **different previous tabular data prediction models only excel on a part of datasets** (see Table 2: excluding Excelformer, the best models vary in different groups). In this paper, Excelformer tries to realize such dominance. Previous works ignored such dataset diversity, and most of them (e.g., TabNet, Node, SAINT) only validated their claims on limited datasets. Similarly, the conclusions in the paper titled *Tabular data: Deep learning is not all you need* may also not be entirely convincing, as it relied on only 11 datasets. Hence, in our paper, **we extensively utilized 130+ diverse datasets, and also presented results categorized by characteristics in Table 2**. See Table 2, one may observe that different previous methods exhibit preferences for distinct datasets, but Excelformer obtains top performances in all the groups.

(iii) **Why propose several components?** Given the diversity of datasets in this field and the no-free-lunch principle, it is challenging to find a component widely applicable to all the dataset types (under default hyper-parameter setting). Following your suggestion, we compare the Excelformer (without data augmentation) against previous models, and found Excelformer (without data augmentation) achieves the best performance (see `Q4 & A4`). However, as an application-oriented research, we prioritize both model performance and user convenience. Thus, in this paper, we introduce those useful components, and we find they can make Excelformer:

* exempt from hyper-parameter tuning (using default hyperparameters is very convenient in application!)

* capable of robust performance across various types of datasets and tasks (see Table 2)

We believe that the purpose of AI research is to facilitate practical applications. The combination of versatility and the convenience of not requiring hyperparameter tuning positions Excelformer as an excellent practical tool on tabular data prediction tasks, especially for users with limited computer science proficiency or computational resources.

(iv) **Why XGBoost outperforms CatBoost?**

* Firstly, in our experiments, **XGBoost does not consistently outperform CatBoost**. See Table 3, under default parameters, CatBoost outperforms XGBoost. Additionally, see Table 2, CatBoost performs better than XGBoost in classification and the case of #. Features $\ge$ 16. Besides, CatBoost typically outperforms XGboost in multi-class classification (see Table 7 of the revised version). **Such distinction is due to the diversity of datasets and tasks.**

* A key contribution of CatBoost is reducing the workload for users in terms of feature engineering, visualization, and hyper-parameter tuning. This enables CatBoost to outperform XGBoost when using default parameters. However, after conducting extensive hyperparameter search for GBDTs (see Tables 6 and 7 - we provide more estimator candidates (4096) than previous works), we find that XGboost can outperform CatBoost on over 50% of datasets.

**In summary, Excelformer is an application-oriented research. It achieves robust performance across diverse datasets using default hyperparameters. The proposed components work together and make Excelformer superior on the diversity of tabular data with almost no added cost.**

Reviewer comment (Evaluation):
I would strongly disagree with the use of accuracy for multi-class classification problems. Not only is it not consistent with binary classification problems, but it additionally does not capture the performance with imbalanced datasets.

Author response:
**A6**: ACC is the standard metric for multi-class classification tasks, which is widely used in previous tabular data prediction papers `[2,3,4]`. We try to follow these works so that offer easy reference for followers.

Reviewer comment (Evaluation):
Using reduced rank as the main and only metric of model performance. There are multiple problems with this approach to reporting the results, which keep me from agreeing with the ExcelFormer performance claims:
- On this particular set of datasets DL models already perform on-par with GBDTs in terms of average rank (see FT-Transformer avg. rank), thus win over GBDT
- The degree of improvement (in terms of the task metrics) is not quantifiable from the average rank. Did the ExcelFormer improve upon vanilla FT-Transformer by 10%, 50% in terms of AUC, neg. RMSE, ACC? The magnitude of the improvement is also important.

Author response:
Thank you for your suggestions, and we acknowledge the limitations of mean-rank. We utilized the performance rank following previous works (e.g., `[1]`). Additionally, we compute **the average normalized scores** (introduced in `[2]`) to aggregate the model performances across datasets. The results are presented in Table 2, Table 7, Table 8 in the revised version. The average normalized scores linearly scale the model's performance, providing insights into the performance differences among models beyond just ranking. The definitions are presented in Appendix F.
Interestingly, we observe that average normalized score and performance rank tend to offer consistent evaluations of the models. This consistency may be attributed to the substantial volume of datasets we employed. **Both performance rank and average normalized scores consistently demonstrate that our model outperforms previous works, on various datasets.**

The detailed results are summarized below:

**(1)** The performances on 96 small-scale datasets:

* Binary-class Classification (51%; metric: Rank{$\downarrow$} (Average Normalized AUC {$\uparrow$}); best are in bold):

| Model  | Excelformer | FTT (t)   | XGb (t)  | Cat (t)| MLP (t) | DCNv2 (t)     | AutoInt (t)    | SAINT (t)    | TransTab (d) | XTab (d)               | TabPFN (t)         |
|-|-|--|-|-|-|--|--|--|--|--|--|
| Setting: Hid-Mix (d) | **3.88 (0.79)**  | 4.88 (0.74)   | 5.97 (0.63)  | 5.77 (0.65)   | 6.61 (0.60)    | 6.38 (0.57)       | 6.63 (0.56)            | 6.07 (0.64)  | 6.31 (0.64)   | 9.50 (0.24)    | 4.01 (0.78) |
| Setting: Mix Tuned   | **3.78 (0.80)**      | 4.91 (0.73)     | 5.95 (0.62)  | 5.79 (0.64)   | 6.60 (0.59)    | 6.39 (0.56)           | 6.71 (0.56)            | 6.10 (0.63)            | 6.37 (0.63)   | 9.46 (0.24)             | 3.95 (0.78)   |

* Regression (49\%; metric: Rank{$\downarrow$} (Average Normalized nRMSE {$\uparrow$}); best are in bold):

| Model | Excelformer  | FTT (t)  | XGb (t)  | Cat (t)   | MLP (t) | DCNv2 (t)  | AutoInt (t) | SAINT (t) | TransTab (d)  | XTab (d)  |
|-|-|--|--|--|--|-|--|--|-|--|
| Setting: Hid-Mix (d) | **3.81 (0.81)**  | 4.45 (0.78)  | 3.43 (0.83)      | 4.26 (0.80)  | 4.64 (0.74)   | 6.26 (0.52)  | 5.53 (0.65)  | 5.64 (0.66)  | 8.21 (0.33)   | 8.79 (0.20)   |
| Setting: Mix Tuned   | **3.17 (0.86)**   | 4.49 (0.78)    | 3.53 (0.82)       | 4.28 (0.81)     | 4.74 (0.74)   | 6.32 (0.52)    | 5.68 (0.65)   | 5.72 (0.65)   | 8.23 (0.33)  | 8.83 (0.19)    |

**(2)** On 21 larger-scale datasets, we compute the **normalized average scores of ACC, AUC, and nRMSE ($\uparrow$)** on multi-class classification datasets, binary-class classification datasets, and regression datasets, respectively. **It is evident that Excelformer performs best.**

| Model (default hyperparameters)  | binclass    | regression  | multiclass|
|-|-|-|-|
| XGboost (d)  | 0 $\pm$ 0    | 0.470$\pm$0.400  | 0.218$\pm$ 0.400 |
| Catboost (d) | 0.977$\pm$0.034  | 0.286$\pm$0.291  | 0.280$\pm$0.408  |
| FTT (d)   | 0.807$\pm$0.397  | 0.613$\pm$0.313  | 0.763$\pm$0.163  |
| ExcelFormer w/ Feat-Mix  (d) | 0.982$\pm$0.022  | 0.513$\pm$0.422  | 0.791$\pm$0.143  |
| ExcelFormer w/ Hid-Mix (d) | **0.976**$\pm$0.030 | **0.825**$\pm$0.279 | **0.990**$\pm$0.020 |

| Model (hyperparameter tuned) | binclass | regression| multiclass |
|-|-|-|-|
| XGboost (t) | 0.409$\pm$0.424  | 0.756$\pm$0.285  | 0.258$\pm$0.241  |
| Catboost (t) | 0.281$\pm$0.365  | 0.276$\pm$0.385  | 0.095$\pm$0.135  |
| FT-T (t)   | 0.316$\pm$0.387  | 0.428$\pm$0.398  | 0.596$\pm$0.425  |
| ExcelFormer (Mix Tuned)| 0.526$\pm$0.413  | **0.865**$\pm$0.150 | 0.735$\pm$0.286  |
| ExcelFormer (Fully Tuned) | **0.777**$\pm$0.393 | **0.865**$\pm$0.203 |**0.957**$\pm$0.085 |

**(3)** Following the suggestions of the reviewer `J7k8`, we also provide the results on 16 **additional multi-class classification datasets** (refer to Table 7 in the revised version), and **both performance rank and average normalized ACC indicate that Excelformer performs best**:

| | XGb (t)       | Cat (t)  | FTT(t)  | MLP(t) | DCNv2(t)   | AutoInt(t)  | SAINT(t) | Excelformer(t)  |
|-|-|-|--|-|-|-|--|--|
| Rank($\pm$std) {$\downarrow$}    | 3.91$\pm$2.28     | 2.53$\pm$1.70     | 5.53$\pm$1.85    | 6.09$\pm$1.88    | 5.97$\pm$1.61    | 4.66$\pm$1.65    | 5.22$\pm$1.41    | **2.09$\pm$1.23** |
| average normalized ACC ($\pm$std) {$\uparrow$} | 0.65$\pm$0.35 | 0.82$\pm$0.27    | 0.39$\pm$0.33   | 0.25$\pm$0.31   | 0.33$\pm$0.35   | 0.50$\pm$0.31   | 0.42$\pm$0.29   | **0.85$\pm$0.24** |

---

References:

`[1]`XTab: Cross-table Pretraining for Tabular Transformers. ICML, 2023

`[2]`Learning hyperparameter optimization initializations. DSAA, 2015

Reviewer comment (Evaluation):
Reference [1] should be considered regarding issues with comparing average ranks of multiple algorithms across multiple datasets.

Author response:
Thank you for your suggestions, and we acknowledge the limitations of mean-rank. We utilized the performance rank following previous works (e.g., `[1]`). Additionally, we compute **the average normalized scores** (introduced in `[2]`) to aggregate the model performances across datasets. The results are presented in Table 2, Table 7, Table 8 in the revised version. The average normalized scores linearly scale the model's performance, providing insights into the performance differences among models beyond just ranking. The definitions are presented in Appendix F.
Interestingly, we observe that average normalized score and performance rank tend to offer consistent evaluations of the models. This consistency may be attributed to the substantial volume of datasets we employed. **Both performance rank and average normalized scores consistently demonstrate that our model outperforms previous works, on various datasets.**

The detailed results are summarized below:

**(1)** The performances on 96 small-scale datasets:

* Binary-class Classification (51%; metric: Rank{$\downarrow$} (Average Normalized AUC {$\uparrow$}); best are in bold):

| Model  | Excelformer | FTT (t)   | XGb (t)  | Cat (t)| MLP (t) | DCNv2 (t)     | AutoInt (t)    | SAINT (t)    | TransTab (d) | XTab (d)               | TabPFN (t)         |
|-|-|--|-|-|-|--|--|--|--|--|--|
| Setting: Hid-Mix (d) | **3.88 (0.79)**  | 4.88 (0.74)   | 5.97 (0.63)  | 5.77 (0.65)   | 6.61 (0.60)    | 6.38 (0.57)       | 6.63 (0.56)            | 6.07 (0.64)  | 6.31 (0.64)   | 9.50 (0.24)    | 4.01 (0.78) |
| Setting: Mix Tuned   | **3.78 (0.80)**      | 4.91 (0.73)     | 5.95 (0.62)  | 5.79 (0.64)   | 6.60 (0.59)    | 6.39 (0.56)           | 6.71 (0.56)            | 6.10 (0.63)            | 6.37 (0.63)   | 9.46 (0.24)             | 3.95 (0.78)   |

* Regression (49\%; metric: Rank{$\downarrow$} (Average Normalized nRMSE {$\uparrow$}); best are in bold):

| Model | Excelformer  | FTT (t)  | XGb (t)  | Cat (t)   | MLP (t) | DCNv2 (t)  | AutoInt (t) | SAINT (t) | TransTab (d)  | XTab (d)  |
|-|-|--|--|--|--|-|--|--|-|--|
| Setting: Hid-Mix (d) | **3.81 (0.81)**  | 4.45 (0.78)  | 3.43 (0.83)      | 4.26 (0.80)  | 4.64 (0.74)   | 6.26 (0.52)  | 5.53 (0.65)  | 5.64 (0.66)  | 8.21 (0.33)   | 8.79 (0.20)   |
| Setting: Mix Tuned   | **3.17 (0.86)**   | 4.49 (0.78)    | 3.53 (0.82)       | 4.28 (0.81)     | 4.74 (0.74)   | 6.32 (0.52)    | 5.68 (0.65)   | 5.72 (0.65)   | 8.23 (0.33)  | 8.83 (0.19)    |

**(2)** On 21 larger-scale datasets, we compute the **normalized average scores of ACC, AUC, and nRMSE ($\uparrow$)** on multi-class classification datasets, binary-class classification datasets, and regression datasets, respectively. **It is evident that Excelformer performs best.**

| Model (default hyperparameters)  | binclass    | regression  | multiclass|
|-|-|-|-|
| XGboost (d)  | 0 $\pm$ 0    | 0.470$\pm$0.400  | 0.218$\pm$ 0.400 |
| Catboost (d) | 0.977$\pm$0.034  | 0.286$\pm$0.291  | 0.280$\pm$0.408  |
| FTT (d)   | 0.807$\pm$0.397  | 0.613$\pm$0.313  | 0.763$\pm$0.163  |
| ExcelFormer w/ Feat-Mix  (d) | 0.982$\pm$0.022  | 0.513$\pm$0.422  | 0.791$\pm$0.143  |
| ExcelFormer w/ Hid-Mix (d) | **0.976**$\pm$0.030 | **0.825**$\pm$0.279 | **0.990**$\pm$0.020 |

| Model (hyperparameter tuned) | binclass | regression| multiclass |
|-|-|-|-|
| XGboost (t) | 0.409$\pm$0.424  | 0.756$\pm$0.285  | 0.258$\pm$0.241  |
| Catboost (t) | 0.281$\pm$0.365  | 0.276$\pm$0.385  | 0.095$\pm$0.135  |
| FT-T (t)   | 0.316$\pm$0.387  | 0.428$\pm$0.398  | 0.596$\pm$0.425  |
| ExcelFormer (Mix Tuned)| 0.526$\pm$0.413  | **0.865**$\pm$0.150 | 0.735$\pm$0.286  |
| ExcelFormer (Fully Tuned) | **0.777**$\pm$0.393 | **0.865**$\pm$0.203 |**0.957**$\pm$0.085 |

**(3)** Following the suggestions of the reviewer `J7k8`, we also provide the results on 16 **additional multi-class classification datasets** (refer to Table 7 in the revised version), and **both performance rank and average normalized ACC indicate that Excelformer performs best**:

| | XGb (t)       | Cat (t)  | FTT(t)  | MLP(t) | DCNv2(t)   | AutoInt(t)  | SAINT(t) | Excelformer(t)  |
|-|-|-|--|-|-|-|--|--|
| Rank($\pm$std) {$\downarrow$}    | 3.91$\pm$2.28     | 2.53$\pm$1.70     | 5.53$\pm$1.85    | 6.09$\pm$1.88    | 5.97$\pm$1.61    | 4.66$\pm$1.65    | 5.22$\pm$1.41    | **2.09$\pm$1.23** |
| average normalized ACC ($\pm$std) {$\uparrow$} | 0.65$\pm$0.35 | 0.82$\pm$0.27    | 0.39$\pm$0.33   | 0.25$\pm$0.31   | 0.33$\pm$0.35   | 0.50$\pm$0.31   | 0.42$\pm$0.29   | **0.85$\pm$0.24** |

---

References:

`[1]`XTab: Cross-table Pretraining for Tabular Transformers. ICML, 2023

`[2]`Learning hyperparameter optimization initializations. DSAA, 2015

Reviewer comment (Experiments):
Limited ablations and comparisons to baselines. The paper proposes a few architectural tweaks for a base transformer model: SPA instead of MHSA, IAI initialization in attention, new FFN block, new nonlinearity, with SPA and IAI highlighted as the more important ones. The section with the ablation is rather short and lacking details regarding the setup, reporting only average rank performance. A more detailed ablation and comparison to the vanilla transformer is needed. For example:
- Transformer (no SPA, IAI, fancy embeddings and GLUs in FFN)
- Transformer + SPA
- Transformer + IAI
- Transformer + SPA + IAI
- ExcelFormer
A subset of datasets with metrics instead of ranks would be enough (see point 1).

Author response:
Thank you for your thoughtful question. We would like to clarify this points from two aspects: the selection of ablation study datasets; presenting more ablation results.

* The selection of tabular datasets. **While some literature claims that GBDTs "dominate" tabular data prediction tasks, it's essential to clarify that this is not an absolute truth.** See Table 2, TabPFN often outperforms GBDTs in classification on small-scale datasets, and GBDTs may perform worse than other DL models when the feature count is lower than 16. Thus, **surpassing GBDTs is not the sole objective in building Excelformer**; beyond defeating GBDTs, we also aim to safeguard the advantageous niche of DL models (we have made some changes to clarify this point in the introduction section; thank you for bringing it to our attention). To simulate real-world applications, our ablation studies are based on all datasets (including those GBDT-best datasets).

* More ablation study results.

**(I)** Key Structure Components

1. Following your suggestion, we conduct additive study to examinate the performances of the key structure components on a vanilla Transformer (Tra). We find that all the components make positive contributions to tabular data prediction tasks, and the contributions are cumulative.

| | Tra | Tra + IAI | Tra + SPA |Tra + SPA + IAI| Excelformer (no data augmentation) |
|-|-|-|-|-|-|
| binclass (Ave Norm AUC, ↑)| 0.232±0.36 | 0.683±0.37|0.289±0.37|0.753±0.33|0.775±0.30|
| regression (Ave Norm nRMSE, ↑) |0.138±0.30| 0.738±0.25| 0.751±0.26| 0.766±0.35 |0.911±0.16|

2. Additionally, we perform an additive study and an ablation study (refer to Fig. 5) to examine the advantages introduced by SPA and IAI in disrupting the rotational invariance property of DL models. We find:

* On the FTT backbone, the addition of SPA and IAI individually to some extent breaks the rotational invariance. 
* On the Excelformer architecture, removing SPA and IAI weakens Excelformer's rotational invariance property.
* **The effects of adding or removing SPA and IAI are cumulative.**

**(II)** Compare Hid-Mix, Feat-Mix, and Resample on MLP:

The results are presented below (in Table 5 of the revised version). We observe that the performances of feature resampling fall between Feat-Mix and Hid-Mix. We notice that each approach excels on different datasets. We will provide more discussions on this aspect in the final version.

|| Resample | Feat-Mix | Hid-Mix | No data augmentation |
|-|-|-|-|-|
| Binclass (Ave Norm AUC, ↑) | 0.576±0.42 | 0.530±0.41 | 0.614±0.41 | 0.436±0.42 |
| regression (Ave Norm nRMSE, ↑)|0.661±0.39|0.629±0.40 | 0.681±0.39 | 0.334±0.42 |

All the additional results have been added into the revised version. Thank you for your constructive suggestion!

Reviewer comment (Experiments):
For the second contribution (novel data augmentations), they could be compared with baselines from literature on tabular data ([2, 3, 4]), where resampling from marginal distributions for a set of columns was shown to be a decent augmentation. Results for the simplest possible setup (like MLP with all features linearly embedded – MLP-LR from [5]) with different augmentation strategies:
- Resample Augmentation
- Feat-Mix
- Hid-Mix
- Feat-Mix + Hid-Mix
would greatly improve the understanding of the efficacy of the proposed augmentations for tabular data.

Author response:
Thank you for your thoughtful question. We would like to clarify this points from two aspects: the selection of ablation study datasets; presenting more ablation results.

* The selection of tabular datasets. **While some literature claims that GBDTs "dominate" tabular data prediction tasks, it's essential to clarify that this is not an absolute truth.** See Table 2, TabPFN often outperforms GBDTs in classification on small-scale datasets, and GBDTs may perform worse than other DL models when the feature count is lower than 16. Thus, **surpassing GBDTs is not the sole objective in building Excelformer**; beyond defeating GBDTs, we also aim to safeguard the advantageous niche of DL models (we have made some changes to clarify this point in the introduction section; thank you for bringing it to our attention). To simulate real-world applications, our ablation studies are based on all datasets (including those GBDT-best datasets).

* More ablation study results.

**(I)** Key Structure Components

1. Following your suggestion, we conduct additive study to examinate the performances of the key structure components on a vanilla Transformer (Tra). We find that all the components make positive contributions to tabular data prediction tasks, and the contributions are cumulative.

| | Tra | Tra + IAI | Tra + SPA |Tra + SPA + IAI| Excelformer (no data augmentation) |
|-|-|-|-|-|-|
| binclass (Ave Norm AUC, ↑)| 0.232±0.36 | 0.683±0.37|0.289±0.37|0.753±0.33|0.775±0.30|
| regression (Ave Norm nRMSE, ↑) |0.138±0.30| 0.738±0.25| 0.751±0.26| 0.766±0.35 |0.911±0.16|

2. Additionally, we perform an additive study and an ablation study (refer to Fig. 5) to examine the advantages introduced by SPA and IAI in disrupting the rotational invariance property of DL models. We find:

* On the FTT backbone, the addition of SPA and IAI individually to some extent breaks the rotational invariance. 
* On the Excelformer architecture, removing SPA and IAI weakens Excelformer's rotational invariance property.
* **The effects of adding or removing SPA and IAI are cumulative.**

**(II)** Compare Hid-Mix, Feat-Mix, and Resample on MLP:

The results are presented below (in Table 5 of the revised version). We observe that the performances of feature resampling fall between Feat-Mix and Hid-Mix. We notice that each approach excels on different datasets. We will provide more discussions on this aspect in the final version.

|| Resample | Feat-Mix | Hid-Mix | No data augmentation |
|-|-|-|-|-|
| Binclass (Ave Norm AUC, ↑) | 0.576±0.42 | 0.530±0.41 | 0.614±0.41 | 0.436±0.42 |
| regression (Ave Norm nRMSE, ↑)|0.661±0.39|0.629±0.40 | 0.681±0.39 | 0.334±0.42 |

All the additional results have been added into the revised version. Thank you for your constructive suggestion!

Reviewer comment (Experiments):
In SPA and Feat-Mix, ExcelFormer uses mutual information. A discussion of how different ways of estimating mutual information compare would be useful, as it seems like a significant detail, but there are no mentions of this in the ablations or the experimental setup.

Author response:
In the implementation, we utilize the scikit-learn package to calculate mutual information. Specifically, for classification tasks, we employ the "feature_selection.mutual_info_classif" function, and for regression tasks, we use the "feature_selection.mutual_info_regression" function. Thank you for your reminder! We have depitcted the implementation in Appendix F.