Paper: Anytime Neural Architecture Search on Tabular Data (ICLR-2024)

Reviewer comment (Experiments):
The search space of NAS-Bench-Tabular (5-layer DNN with different width) is too simple. This search space does not contain most of modern neural network architecture for tabular data [1].

Author response:
**Response:** Existing studies show that deep neural networks (DNNs) can already achieve state-of-the-art performance on tabular data [4, 5], and the technical challenge is to configure DNNs with the right number of layers and hidden layer sizes for each layer [1]. Therefore, we design a DNN-based search space that comprises up to 160,000 candidate DNNs with an extensive collection of different layer configurations.

As shown in the experiments and visualizations in Section 4.1.1 and Appendix C.3, this search space contains a diverse set of architectures with a wide range of parameter sizes, configurations, and performances, which captures the main properties of DNNs for benchmarking NAS approaches on tabular data.

Reviewer comment (Evaluation):
Most of the candidate architectures get similar 97~98% accuracy on Frappe dataset. I wonder how stable the recorded accuracies are.

Author response:
**Response:** We have summarized the statistics of our 160,000 trained architectures on the Frappe dataset in the table below.

| Mean (AUC) | Standard Deviation (AUC) | Maximum (AUC) | Minimum (AUC) |
| - | - | - | - |
| 97.68%     | 0.0021                   | 98.14%        | 85.72%        |

The very low standard deviation (0.0021) indicates that most architectures closely align with the mean AUC. While the significant gap between the maximum AUC (98.14\%) and minimum AUC (85.72\%) suggests that the number of hidden neurons in each layer is crucial in determining the architecture performance.

Reviewer comment (Experiments):
All three datasets are about binary classification tasks. Regression, multi-class classification tasks are needed.

Author response:
**Response:** As suggested, we evaluated ATLAS on additional datasets: three for multi-class classification and two for regression.
The results of multi-class classification are presented in our global response to the _Evaluation of ATLAS on more Datasets_.
The results of the regression are shown in the table below where we have measured the Root Mean Square Error (RMSE).

| **Dataset**         | **XGBoost** | **CatBoost** | **ATLAS** | **Searched Architecture** |
| - | - | - | - | -| 
| California Housing  | 0.139       | **0.131**    | 0.141     | 384-256-256-512           |
| Pol                    | 4.773       | 4.698        | **2.264** | 384-256-128-256           |

ATLAS outperforms both XGBoost and CatBoost on Pol datasets with lower RMSE.
While for the California Housing dataset, ATLAS achieves comparable results with others.

Reviewer comment (Novelty):
In general, performance proxies have the advantage of being data type-agnostic, which is also a benefit of deep learning. Why do we need a TRAILER that specializes in tabular data? And what makes ExpressFlow specializes on tabular data?

Author response:
**Response:** Existing data type-agnostic TRAILERs such as SynFlow do not perform well on tabular datasets as shown in Table 1 of the original paper. Therefore, We are motivated to propose a new TRAILER specifically designed for tabular data.

To this end, we first characterize the trainability and expressivity of existing training-free metrics initially proposed for image data. Then, we propose a new TRAILER tailored for tabular data based on neuron saliency. 
This TRAILER characterizes both properties for DNNs on tabular data (see discussion in Section 3.2 and theoretical analysis in Appendix D.3), by considering the activation value of neurons that are basic units to extract features in DNNs and the derivatives of neurons that indicate the importance of features extracted by neurons.

Reviewer comment (Theory):
In Equation 2, aren’t Successive Halving Algorithm allocates different budget U and candidate K for each round?

Author response:
**Response:** Our ATLAS includes both filtering and refinement phases. For the refinement phase, it uses the Successive Halving Algorithm (SUCCHALF) to identify the best-performing architecture from the $K$ architectures, with each training for at least $U$ epochs.

Indeed, SUCCHALF allocates a different budget of $U * \eta^i$ (with $U$ doubling if $\eta$ = 2) and a number of candidate architectures $\lfloor K/\eta^i \rfloor$ (halving if $\eta$ = 2) for training-based evaluation in the $i$-th round (with i starting from 0). 
For example, in the first round of evaluation, SUCCHALF trains $K$ architectures, with each training for $U$ epochs. 
In the second round, the higher-performing $\lfloor K/\eta\rfloor$ architectures are retained, with each training for $U * \eta$ epochs. This process iterates for $\lfloor\log_{\eta}K\rfloor$ training rounds until one single architecture remains.

Reviewer comment (Experiments):
**Evaluation of ATLAS on more diverse datasets**: This paper contains multiple contributions, starting from the proposal of a new NAS benchmark for tabular data to the NAS algorithm for tabular data. While these are very useful and provide interesting insights, it can also introduce many biases. Typically, one would design an algorithm that works on benchmarks (real ones) that the community already uses in their research. Firstly designing a NAS benchmark for tabular data and then proposing a method and evaluating it only on that benchmark does not guarantee the same behavior on the real benchmarks. Therefore, I would really be interested to see how ATLAS works on commonly used OpenML tabular benchmarks (e.g. the ones from [1], Table 9), that are more diverse and lie more into the small-data regime. If the authors provide competitive results on those datasets, I am willing to increase my score.

Author response:
**Response:** As suggested, we have evaluated our ATLAS on more datasets from the open-source OpenML AutoML Benchmark, and the results are presented in our global response to the _Evaluation of ATLAS on more Datasets_.

Reviewer comment (Experiments):
**Comparison with conventional tree-based models**: To assess the usefulness of ATLAS in practice, I think the authors should compare the best found network with conventional tree-based models, as it is done in [1], [2] and [3], for instance. An easy and fair experimental setup would be: (1) Run ATLAS for a given time budget to find architecture **x**; (2) Train and evaluate **X** on the full fidelity (if not already evaluated in (1)); (3) Run tree-based models for *t* time (total runtime of (1) + runtime of (2)).

Author response:
**Response:** As suggested, we compared ATLAS with XGBoost, assessing their AUC on the Frappe dataset.

Our ATLAS, supporting anytime NAS for tabular data, can operate within a pre-defined time budget. 
We first set a time budget to run ATLAS to select a well-performing architecture. Then, an additional time budget was allocated to fully train and evaluate this selected architecture. 
We managed both time budgets to ensure the total time budget was around one hour.

For XGBoost, we also fixed the overall time budget for one hour and tuned the hyperparameter of XGBoost based on training. We employed the same hyperparameter search space as defined in [1] (Table 6).

The results shown in the Table below demonstrate that ATLAS can identify architectures that outperform XGBoost on the Frappe datasets. This confirms the efficacy of ATLAS in finding well-configured DNNs, i.e., determining the appropriate number of hidden neurons for each layer.

| **Dataset** | **#Samples/#Columns** | **XGBoost** | **ATLAS**  | 
| - | - | - | - | 
| Frappe      | 288,609 / 10          | 97.46%      | **98.03%** |

Reviewer comment (Experiments):
**Comparison to SOTA for DL for tabular data**: A thorough comparison to neural networks that achieve SOTA for tabular data is necessary to highlight the usefulness of NAS in this domain. Similar to the point above I would recommend the authors to compare to TabPFN [3]. If the time permits, I would recommend the authors to run ATLAS on the same settings the TabPFN authors used to obtain Figure 5 in their paper.

Author response:
**Response:** Our ATLAS differs theoretically from TabPFN. 
TabPFN is a Prior-Data Fitted Network (PFN) capable of approximating probabilistic inference in a single forward pass, it has two limitations: 1) it requires extensive offline training (e.g., 20 hours on 8 GPUs as reported in [3]), and 2) it only scales effectively to small datasets and performs worse when categorical features are presented.

In contrast, our ATLAS employs both training-free and training-based architecture evaluation techniques. 
When compared to TabPFN, ATLAS demonstrates robust performance across diverse datasets ranging from small to large and consistently performs well when facing either numerical or categorical features, as evidenced in Figure 6 of the original paper.

Table 5 in the TabPFN paper [3] presents the mean AUC of the TabPFN across all 18 datasets from the OpenML-CC18 Benchmark. Due to time constraints, we have evaluated our ATLAS on only eight datasets. For each dataset, we have also measured the total time (Total Cost) it takes to both search for the higher-performing architecture and then train this architecture to evaluate its performance, as shown in our global response to the _Evaluation of ATLAS on more Datasets_.
The results illustrate ATLAS is highly efficient and is competitive with methods detailed in Table 5 of the TabPFN paper.

Reviewer comment (Writing):
It would be great if the authors follow the already used nomenclature when referring to zero-cost proxies, and not rename them to TRAILERs.

Author response:
**Response:** As suggested, we will refer the TRAILERs to zero-proxies.

Reviewer comment (Evaluation):
In Section 2, third paragraph, it is written "The performance obtained by the training-based architecture evaluation approaches is accurate", however this is not always the case, especially when considering proxy models (less layers, channels, etc.) or performance evaluation based on the one-shot model weights (e.g. as in DARTS).

Author response:
**Response:** As suggested, we will rephrase the sentence to "The performance obtained by the training-based architecture evaluation approaches is typically more accurate".

Reviewer comment (Novelty):
Most of blackbox algorithms are also anytime algorithms, and they are widely used for NAS as well. Therefore, the claim that ATLAS is the first NAS anytime algorithm (in abstract) for tabular data is wrong considering that the architectural parameters have been optimized using these methods (e.g. Bayesian Optimization) and could easily be used to optimize architectural hyperparameters for tabular data networks as well. Ideally, one wants an optimization method that is modality agnostic. ATLAS is the first anytime algorithm tailored for tabular data because of *ExpressFlow* is designed for tabular data. And yes, *ExpressFlow* is the first zero-cost proxy designed for tabular data, so I would emphasize that instead.

Author response:
**Response:** Thank you for your valuable suggestions. We will first rephrase our claims to state 'ATLAS is the first anytime algorithm tailored for tabular data.' Second, we will emphasize that 'ExpressFlow is the first zero-cost proxy designed for tabular data' in our paper.

Reviewer comment (Experiments):
Is ATLAS performant on datasets with less than 1000 data points?

Author response:
**Response:** As suggested, we have evaluated our ATLAS on more datasets from the open-source OpenML AutoML Benchmark, and the results are presented in our global response to the _Evaluation of ATLAS on more Datasets_.

Reviewer comment (Evaluation):
How many data points were used to compute the rank correlation in Table 1?

Author response:
**Response:** In Table 1 of our paper, we present the correlation between each training-free evaluation metric score and the architecture's actual performance (AUC).  This analysis uses 160,000 architectures for the Frappe and Diabetes datasets and 10,000 for the Criteo dataset.

Reviewer comment (Experiments):
A limitation of the paper is the selection of benchmark datasets that may not comprehensively represent the diversity of real-world tabular data. The features of the chosen datasets—Frappe, Diabetes, and Criteo—are relatively low in dimensionality (10, 43, and 39 features, respectively). This narrow scope could potentially limit the generalizability of the study's findings. To convincingly argue the efficacy of ATLAS across various scenarios, it would be beneficial to test it on a wider range of datasets with varying feature dimensions, complexity, and domain-specific challenges. The current dataset selection might not fully challenge the capability of ATLAS to handle higher-dimensional and more complex tabular datasets that are commonly found in practice.

Author response:
The datasets utilized in our work are real-world tabular data. Specifically, we select three datasets from different application domains: app recommendation, healthcare analytics, and CTR (e-commerce) prediction. The statistics of the datasets are summarized in the Table below:

| Dataset  | # Class | # Samples  | # Columns | # Features | Task                 |
| -------- | ------- | ---------- | --------- | ---------- | -------------------- |
| Frappe   | 2       | 288,609    | 10        | 5,382      | App Recommendation   |
| Diabetes | 2       | 101,766    | 43        | 369        | Healthcare Analytics |
| Criteo   | 2       | 45,840,617 | 39        | 2,086,936  | CTR Prediction       |

Notably, for the three datasets, there are 10, 43, and 39 columns, corresponding to 5,382, 369, and  2,086,936 unique numerical and categorical features respectively, which are high-dimensional.

To further validate the generalizability and effectiveness of our proposed ATLAS, we have conducted experiments on an additional eight datasets from the open-source OpenML AutoML Benchmark.

The evaluation results, as presented in our global response to the _Evaluation of ATLAS on More Datasets_, confirm the effectiveness and generalizability of our proposed ATLAS.

Reviewer comment (Experiments):
The missing significant baselines, particularly established methods such as XGBoost and various Transformer-based models like TabTransformer and FTTransformer. Including these baselines is crucial for a comprehensive comparative analysis, especially to substantiate the necessity and superiority of NAS in the domain of tabular data. By neglecting to compare ATLAS against these well-known and widely-used methods, the paper misses an opportunity to demonstrate the practical advantage of NAS for tabular data over more traditional, yet powerful, approaches. This comparative analysis is essential to persuade the research community and industry practitioners of the added value that ATLAS and, more broadly, NAS methods may provide in tabular data applications.

Author response:
We would like to clarify that the scope of this paper is to propose a novel anytime NAS approach tailored for tabular data, rather than to design a new architecture that outperforms established architectures such as TabTransfer or FT-Transfermor. 
As a result, we mainly pick baselines from the state-of-the-art NAS approaches designed for tabular data. 
Specifically, we utilize TabNAS [1] and training-based NAS approaches, i.e., RE-NAS, for comparison as shown in Table 6 and discussed in Section 4.3 of the original paper.

To further demonstrate the practical advantages of our ATLAS, we have measured its balanced accuracy on two datasets from the OpenML AutoML Benchmark and compared ATLAS against XGBoost, TabTransformer, and FTTransformer. The results shown in the table below indicate that the selected architecture of ATLAS is comparable to both transformer-based architectures and XGBoost.

| **Dataset** | **#Samples/#Columns** | **XGB** | **TabTransformer** | **FTTransformer** | **ATLAS** | **Searched Architecture** |
| ----------- | --------------------- | ------- | ------------------ | ----------------- | --------- | ------------------------- |
| Adult       | 48842 / 15            | 79.82   | 78.09              | 78.42             | 78.40     | 384-256-256-512           |
| Bank        | 45211 / 17            | 72.66   | 78.57              | 78.37             | 78.32     | 384-256-256-512           |

Both TabTransformer and FTTransformer are designed to capture complex relationships in tabular data through self-attention mechanisms, and thus they perform consistently well on two datasets.
Notably, our ATLAS effectively determines the number of hidden neurons for each layer of an MLP, thereby making it competitive with other methods. The experimental results align with conclusions drawn from FTTransformer [2]: specific tuning can make simple models like MLPs competitive. 

We hope our responses above have addressed your concerns and can improve your evaluation of our work.

[1] TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets [Yang et al., NeurIPS 2022].

[2] Revisiting Deep Learning Models for Tabular Data [Gorishniy et al., NeurIPS 2021].

Reviewer comment (Experiments):
Could you comment on the applicability and expected behavior of the proposed method on other kinds of architecture search spaces other than the search space used in this paper?

Author response:
**Response:** Our proposed method, ExpressFlow, is transferable and can be applied to other search spaces, such as NAS-BENCH-101 and NAS-BENCH-201 [4], which are designed for vision tasks. 

However, ExpressFlow is specifically tailored for tabular data, as it is based on neuron saliency that captures the complex and non-intuitive relationships among input features in this data type. 

Consequently, while demonstrating higher correlation values on tabular datasets (as shown in Table 1 of the original paper), ExpressFlow achieves only the second-highest average rank in terms of correlation value for image data, as illustrated in Table 15 of Appendix G.1.