Paper: AdapTable: Test-Time Adaptation for Tabular Data via Shift-Aware Uncertainty Calibrator and Label Distribution Handler (ICLR-2024)

Reviewer comment (Experiments):
This paper does not investigate and justify (semantically and experimentally) the advantages of the proposed post-hoc output calibration method compared with traditional model calibration methods such as isotonic regression calibration and Platt calibration.

Author response:
The calibration method proposed in the 'AdapTable' paper, referred to as the shift-aware post-hoc uncertainty calibrator, offers several advantages over traditional calibration methods like isotonic regression calibration and Platt calibration, especially in the context of handling distribution shifts in tabular data. Here are the key advantages:

**Shift-Awareness:** One of the main innovations of the calibration method in 'AdapTable' is its awareness of distribution shifts. Traditional methods like isotonic regression and Platt calibration are typically applied in a setting where the training and test data are assumed to be drawn from the same distribution. In contrast, the shift-aware calibrator in **'AdapTable' is specifically designed to handle distribution shifts,** making it more suitable for real-world scenarios where the test data might differ significantly from the training data.

**Graph Neural Network Integration:** The calibrator in 'AdapTable' utilizes a Graph Neural Network (GNN) to **understand the relationships between different features (columns) of the data.** This approach is particularly beneficial for tabular data, which often comprises a mix of numerical and categorical features, and each feature encompasses a distinctive meaning, unlike pixels within vision domains. Traditional methods do not typically leverage such relational information between features, which can be crucial for accurate calibration in complex datasets.

**Handling Underconfidence and Overconfidence:** The 'AdapTable' method **addresses both underconfidence and overconfidence in predictions,** a significant improvement over traditional methods. Platt calibration, for instance, is often more focused on correcting overconfident predictions and may not be as effective in dealing with underconfidence.

To address the concerns of the reviewer, we’ve additionally conducted an ablative study by exchanging the calibrator to a. Platt calibrator *(PC)* and b. Isotonic regression *(IR)* in the table below. It is empirically shown that such calibration methods such as Platt Calibration, and Isotonic Regression, denoted as their abbreviations, show suboptimal performance when jointly used with our label shift handler. (DIABETES indicates DIABETES READMISSION dataset.)

| (Acc. %)  | HELOC      | ANES       | DIABETES   | CMC        | MFEAT-PIXEL | DNA        |
| ---- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- |
| Unadapted | 47.0 ±1.6 | 79.3 ±0.2 | 61.3 ±0.1| 53.5 ±1.4 | 96.4 ±0.2 | 91.4 ±1.2 |
| PC   | 61.6 ± 1.3 | 79.2 ± 0.2 | 61.4 ± 0.3 | 52.1 ± 1.5 | 96.4 ± 0.3  | 91.4 ± 0.8 |
| IR   | 61.3 ± 1.7 | 79.2 ± 0.2 | 61.0 ± 0.4 | 51.9 ± 1.6 | 96.0 ± 0.3  | 91.3 ± 0.8 |
| **Ours** | **64.5 ± 0.3** | **79.6 ± 0.1** | **61.7 ± 0.0** | **55.7 ± 2.0** | **97.8 ± 0.2**  | **95.0 ± 0.5** |

Reviewer comment (Theory):
It is not clear how the target label distribution output by the calibration model is guaranteed to be correct.

Author response:
Ensuring the accuracy of the target label distribution in our 'AdapTable' framework involves two key strategies:

**De-biasing Source Label Distribution:** Grounded in Bayes' theorem, we address the inherent bias in the prediction logits of the source model towards the source label distribution. This is achieved by dividing the prediction logits by the source label distribution, which is obtained prior to the model's deployment. This step effectively debiases the predictions, making them more representative of the actual target distribution rather than being overly influenced by the source data characteristics.

**Iterative Refinement Strategy:** The estimation and adjustment of the target label distribution in 'AdapTable' is an iterative process. As the model encounters more test data, it continually refines its understanding and estimation of the target label distribution. This iterative refinement is key to gradually improving the accuracy of the label distribution estimation, ensuring that the model adapts more effectively to the target domain over time.

With these strategies, we are able to estimate the target label distribution with a considerable degree of accuracy. The empirical performance of this approach is demonstrated in our paper, particularly in **Figure 5** and **Figure 9** in our revised paper, where we plot the Jensen-Shannon divergence of distributions before and after applying our label distribution handler. This plot illustrates the effectiveness of our method in aligning the model’s predictions with the actual label distribution in the target domain, showcasing the practical utility of our approach in handling label distribution shifts in real-world scenarios.

We sincerely appreciate the reviewer again for the thoughtful comments.

Reviewer comment (Novelty):
What are the advantages of the calibration method proposed in this paper compared to the traditional calibration method such as isotonic regression calibration and Platt calibration?

Author response:
The calibration method proposed in the 'AdapTable' paper, referred to as the shift-aware post-hoc uncertainty calibrator, offers several advantages over traditional calibration methods like isotonic regression calibration and Platt calibration, especially in the context of handling distribution shifts in tabular data. Here are the key advantages:

**Shift-Awareness:** One of the main innovations of the calibration method in 'AdapTable' is its awareness of distribution shifts. Traditional methods like isotonic regression and Platt calibration are typically applied in a setting where the training and test data are assumed to be drawn from the same distribution. In contrast, the shift-aware calibrator in **'AdapTable' is specifically designed to handle distribution shifts,** making it more suitable for real-world scenarios where the test data might differ significantly from the training data.

**Graph Neural Network Integration:** The calibrator in 'AdapTable' utilizes a Graph Neural Network (GNN) to **understand the relationships between different features (columns) of the data.** This approach is particularly beneficial for tabular data, which often comprises a mix of numerical and categorical features, and each feature encompasses a distinctive meaning, unlike pixels within vision domains. Traditional methods do not typically leverage such relational information between features, which can be crucial for accurate calibration in complex datasets.

**Handling Underconfidence and Overconfidence:** The 'AdapTable' method **addresses both underconfidence and overconfidence in predictions,** a significant improvement over traditional methods. Platt calibration, for instance, is often more focused on correcting overconfident predictions and may not be as effective in dealing with underconfidence.

To address the concerns of the reviewer, we’ve additionally conducted an ablative study by exchanging the calibrator to a. Platt calibrator *(PC)* and b. Isotonic regression *(IR)* in the table below. It is empirically shown that such calibration methods such as Platt Calibration, and Isotonic Regression, denoted as their abbreviations, show suboptimal performance when jointly used with our label shift handler. (DIABETES indicates DIABETES READMISSION dataset.)

| (Acc. %)  | HELOC      | ANES       | DIABETES   | CMC        | MFEAT-PIXEL | DNA        |
| ---- | ---------- | ---------- | ---------- | ---------- | ----------- | ---------- |
| Unadapted | 47.0 ±1.6 | 79.3 ±0.2 | 61.3 ±0.1| 53.5 ±1.4 | 96.4 ±0.2 | 91.4 ±1.2 |
| PC   | 61.6 ± 1.3 | 79.2 ± 0.2 | 61.4 ± 0.3 | 52.1 ± 1.5 | 96.4 ± 0.3  | 91.4 ± 0.8 |
| IR   | 61.3 ± 1.7 | 79.2 ± 0.2 | 61.0 ± 0.4 | 51.9 ± 1.6 | 96.0 ± 0.3  | 91.3 ± 0.8 |
| **Ours** | **64.5 ± 0.3** | **79.6 ± 0.1** | **61.7 ± 0.0** | **55.7 ± 2.0** | **97.8 ± 0.2**  | **95.0 ± 0.5** |

Reviewer comment (Theory):
How is the target label distribution ensured (or ensured with a certain probability) to be correct?

Author response:
Ensuring the accuracy of the target label distribution in our 'AdapTable' framework involves two key strategies:

**De-biasing Source Label Distribution:** Grounded in Bayes' theorem, we address the inherent bias in the prediction logits of the source model towards the source label distribution. This is achieved by dividing the prediction logits by the source label distribution, which is obtained prior to the model's deployment. This step effectively debiases the predictions, making them more representative of the actual target distribution rather than being overly influenced by the source data characteristics.

**Iterative Refinement Strategy:** The estimation and adjustment of the target label distribution in 'AdapTable' is an iterative process. As the model encounters more test data, it continually refines its understanding and estimation of the target label distribution. This iterative refinement is key to gradually improving the accuracy of the label distribution estimation, ensuring that the model adapts more effectively to the target domain over time.

With these strategies, we are able to estimate the target label distribution with a considerable degree of accuracy. The empirical performance of this approach is demonstrated in our paper, particularly in **Figure 5** and **Figure 9** in our revised paper, where we plot the Jensen-Shannon divergence of distributions before and after applying our label distribution handler. This plot illustrates the effectiveness of our method in aligning the model’s predictions with the actual label distribution in the target domain, showcasing the practical utility of our approach in handling label distribution shifts in real-world scenarios.

We sincerely appreciate the reviewer again for the thoughtful comments.

Reviewer comment (Experiments):
The experiments in this paper are insufficient. There are 15 datasets in the TableShift benchmark; however, only three of them are considered in the experiments. This makes the results of this paper unconvincing.

Author response:
Your concern regarding the selection of only three datasets from the TableShift benchmark for our experiments is indeed valid and warrants a detailed explanation. In 'AdapTable', our focus was on demonstrating the method's effectiveness in real-world scenarios, and the choice of datasets was guided by these objectives.

**Representativeness:**

The datasets HELOC, Diabetes Readmission, and ANES are foundational and widely recognized in their respective data science domains. Their significant citation counts – over 2000 for HELOC and DIABETES, and more than 8000 for ANES – are indicative of their broad acceptance and extensive utilization in research. 

**Realistic Scenario and Practical Relevance:**

Test-time adaptation (TTA) is particularly pertinent when domain shifts are prevalent during the deployment stage of models, thus we selected datasets that target a realistic scenario in which (1) deployment of pretrained models, and (2) domain shift during test-time, is likely. Each of the selected datasets – HELOC, DIABETES, and ANES – represents a distinct and realistic application of machine learning in the fields of healthcare, finance, and political science, where domain shifts such as regions may directly affect the underlying distribution of data.

**Additional Experiments for Broader Validation:**

Acknowledging the potential limitation of using a relatively small number of datasets for validation, we have conducted additional experiments within the TableShift benchmark. These experiments include datasets varying in size, such as ASSISTments with 2,667,776 observations and Childhood Lead with 27,499 observations. For these experiments, we used the same backbone model (MLP) as in our initial studies. This extension of our experimental evaluation further demonstrates the adaptability and effectiveness of 'AdapTable' across a broader range of scenarios and dataset sizes.

To verify the validity of ours in various datasets in TableShift, we conduct extra experiments in TableShift datasets (BRFSS BLOOD PRESSURE, NHANES LEAD, and ASSISMENTS) with MLP backbone, and the results are in the table in the following comment. We confirm that our method is not tailored for some specific TableShift datasets.

Reviewer comment (Evaluation):
The performance improvement on ANES and Diabetes ReadMission is relatively weak in the Supervised setting (which is the setting with the overall best performance). This makes the proposed method weak.

Author response:
As discernible from the main paper text or algorithm names, architectures labeled 'supervised' in the Table 1 and Table 2 are currently not adopted in other modalities but are still embraced in the tabular domain due to their performance, representing relatively classical algorithms that diverge from modern deep learning practices. Furthermore, a comparison focusing solely on the highest performances across the entire table indicates that these methods, as commonly known, continue to be adopted due to their ability to surpass deep learning performance (regardless of the architecture of deep learning models). When dealing with tabular data in a data-wise manner, many issues arise from this, the difficulty in utilizing recent deep learning-based research analyses and their derivatives, including test time adaptation. However, our approach differs from other TTA methods by not requiring updates or gradients from the main model. Our method is capable of orthogonal utilization even with algorithms that are SoTA but do not seamlessly align with deep learning methodologies. Despite its amount, this has resulted in performance improvements in numerous cases, demonstrating that using our method alongside classical algorithms is not only feasible but also a rational choice.

Reviewer comment (Experiments):
The hyper-parameter of the proposed method lacks thoughtful discussion. For example, the alpha in the label distribution handler determines how quickly the model can adapt to the latest label distribution, which may seriously affect the performance.

Author response:
To assess the robustness of AdapTable against hyperparameter configurations, we conduct an exhaustive hyperparameter sensitivity analysis covering all test-time parameters, including the smoothing factor $\alpha$, low uncertainty quantile $q_{\text{low}}$, and high uncertainty quantile $q_{\text{high}}$. Specifically, in our experiments utilizing MLP on HELOC dataset, we perform hyperparameter optimization by varying one parameter while keeping the others fixed at the identified optimal setting, *i.e.*, $(\alpha, q_{\text{low}}, q_{\text{high}}) = (0.1, 0.25, 0.75)$. Notably, as Figure 7 exhibits, our findings reveal that the adaptation performance remains insensitive to alterations in all three types of hyperparameters, particularly when varying $q_{\text{low}}$, demonstrating minimal performance fluctuations. Furthermore, for the smoothing factor $\alpha$ and high quantile $q_{\text{high}}$, we pinpoint sweet spots at $[0, 0.2]$ and $[0.5, 0.6]$, respectively. This observation underscores the adaptability of our approach, allowing flexible hyperparameter selection and demonstrating generalizability across diverse test conditions. This stands in stark contrast to the hyperparameter sensitivity exhibited by the tent, as depicted in Figure 8 in our revised appendix of paper. Notably, regardless of an extensive hyperparameter search in the tabular domain for the tent, the performance post-adaptation fails to surpass the unadapted performance, as evidenced by our main table experiment results in Table 2 and Table 3.

Reviewer comment (Theory):
This method calibrates the logits with sample-wise temperature and estimated label distribution, which can only handle the label distribution shift problems rather than covariate shift problems.

Author response:
While our approach does not explicitly target covariate shifts, **the nature of the shift in tabular data differs from that in visual data, where a specific corruption can easily be applied to samples while preserving class information.**

In most cases in tabular data, a shift in input space is highly correlated with the shift in the target label  - as shown in the table in the following comment. For instance, in a tabular dataset concerning patient and health information, a shift in the age range of patients directly affects other features and output labels. Our method also deals with covariate shifts by modeling unobserved shifts through the interdependence of shift values during shift-aware calibration. Additionally, by leveraging GNNs to learn the dependence of each column’s shifts, our approach demonstrates superior performance even in synthetic shift scenarios such as random noise or column missing, which are label-shift-free, highlighting its effectiveness in handling covariate shifts.

Reviewer comment (Presentation):
Minor issue: The left sub-figures in Figures 2 and 5 contain black borders.

Author response:
We sincerely thank you for letting us resolve this issue. We update Figure 2 and Figure 5 without black borders in our new draft.

Reviewer comment (Experiments):
The authors should explain why only three datasets are adopted in the experiments and why these three datasets are selected.

Author response:
Your concern regarding the selection of only three datasets from the TableShift benchmark for our experiments is indeed valid and warrants a detailed explanation. In 'AdapTable', our focus was on demonstrating the method's effectiveness in real-world scenarios, and the choice of datasets was guided by these objectives.

**Representativeness:**

The datasets HELOC, Diabetes Readmission, and ANES are foundational and widely recognized in their respective data science domains. Their significant citation counts – over 2000 for HELOC and DIABETES, and more than 8000 for ANES – are indicative of their broad acceptance and extensive utilization in research. 

**Realistic Scenario and Practical Relevance:**

Test-time adaptation (TTA) is particularly pertinent when domain shifts are prevalent during the deployment stage of models, thus we selected datasets that target a realistic scenario in which (1) deployment of pretrained models, and (2) domain shift during test-time, is likely. Each of the selected datasets – HELOC, DIABETES, and ANES – represents a distinct and realistic application of machine learning in the fields of healthcare, finance, and political science, where domain shifts such as regions may directly affect the underlying distribution of data.

**Additional Experiments for Broader Validation:**

Acknowledging the potential limitation of using a relatively small number of datasets for validation, we have conducted additional experiments within the TableShift benchmark. These experiments include datasets varying in size, such as ASSISTments with 2,667,776 observations and Childhood Lead with 27,499 observations. For these experiments, we used the same backbone model (MLP) as in our initial studies. This extension of our experimental evaluation further demonstrates the adaptability and effectiveness of 'AdapTable' across a broader range of scenarios and dataset sizes.

To verify the validity of ours in various datasets in TableShift, we conduct extra experiments in TableShift datasets (BRFSS BLOOD PRESSURE, NHANES LEAD, and ASSISMENTS) with MLP backbone, and the results are in the table in the following comment. We confirm that our method is not tailored for some specific TableShift datasets.

Reviewer comment (Evaluation):
The running time of the proposed method should be reported for both training and evaluation.

Author response:
In order to show the efficiency of the proposed AdapTable, we perform a computational efficiency comparison between test-time adaptation baselines and AdapTable in **Figure 6**. The averaged adaptation time is calculated by averaging adaptation time over all test instances in CMC dataset corrupted by Gaussian noise. We find that the adaptation time of AdapTable ranks third among eight TTA methods, by showcasing its computational tractability. Furthermore, we observe that our approach requires significantly less adaptation time compared to TTA baselines such as TTT++[1], SAM[2], EATA[3], and SAR[4], despite constructing shift-aware graph and incorporating a single forward pass for GNN with negligible extra cost of adjusting output label estimation are required. This can be attributed to the fact that the graph we generate places each column as a node, resulting in a graph of a very small scale -- typically ranging from tens to hundreds of nodes. This minimizes the cost of message passing in GNN forward process, while other baselines iterate through multiple adaptation steps with forward and backward processes, leading to increased computational expenses. 

Furthermore, we also provide GNN post-training time of AdapTable under different scales in Table 5, encompassing small-scale (CMC, MLP), medium-scale (HELOC, FT-Transformer[5]), and large-scale (Diabetes Readmission, TabNet[6]). GNN post-training requires only a few seconds for small- and medium-scale settings, and notably, it remains negligible, even in our largest experimental setting.

We sincerely appreciate the reviewer again for the thoughtful comments.


[1] Yuejiang Liu, et al. "TTT++: When does self-supervised test-time training fail or thrive?" NeurIPS 2021.

[2] Pierre Foret, et al. "Sharpness-aware minimization for efficiently improving generalization" ICLR 2021.

[3] Niu, Wu, Zhang, et al. "Efficient test-time model adaptation without forgetting" ICML 2022.

[4] Niu, Wu, Zhang, et al. "Towards stable test-time adaptation in dynamic wild world" ICLR 2023.

[5] Yury Gorishniy, et al. "Revisiting deep learning models for tabular data" NeurIPS 2021.

[6] Sercan Ö Arik, et al. "TabNet: Attentive interpretable tabular learning" AAAI 2021.

Reviewer comment (Novelty):
Although the proposal can improve the performance, it is mainly a combination of some existing techniques. There is neither good theoretical analysis nor much inspiration, thus, the novelty and contribution are limited. From my personal point of view, I don’t really appreciate papers that improve performance by integrating multiple tricks and existing methods.

Author response:
We conducted extensive experiments to understand the characteristics of deep tabular learned models and the features of domain shifts in the tabular data. Our findings can be summarized as follows: 
1) Deep-learned models for tabular data are often observed to be uncalibrated in terms of uncertainty, exhibiting an overall underconfident nature, in contrast to the widely recognized overconfident behavior of vision domains. (This discovery, especially considering that prior techniques in TTA and domain adaptation predominantly stemmed from vision-based research with many early methods relying on smoothness assumptions like EM and pseudo-labeling, underscores the value of this work.) 
2) Tabular domain shifts, unlike in the vision domain as well, affect the actual output label in most cases, resulting in discernible differences in label distributions between training and testing. While methods addressing label distribution shifts in vision domains have emerged recently, the task itself simulates situations where the output label distribution is adjusted without the cases that the shifts affect the output label, as seen in scenarios like Cifar-corrupted or ImageNet-corrupted datasets. It is crucial to emphasize the distinct nature of tabular data, where shifts in input (covariate) directly influence label and label distribution shifts, prompting the proposal of observation-based, tabular-specific modules capable of handling these shifts. 
3) When utilizing label distribution shift handlers in deep learning models, the effectiveness of the label distribution handler amplifies when the output of the deep learning model is well-calibrated in terms of uncertainty, which is a relatively well-known fact than previous ones. This statement holds in tabular settings, where this effect appears to be substantial. In our revised paper, **Table 1** indicates that if the source model is perfectly calibrated by increasing the confidence for correct samples while decreasing the confidence for incorrect samples, our label distribution handler leads to a remarkable improvement in performance. These interconnected observations emphasize the need for a calibrator tailored for tabular data and a unique handler to address shifts specific to the tabular domain. This elucidates the essential interdependence between both in addressing the nuanced shifts observed in tabular datasets.

Reviewer comment (Theory):
Will the introduction of GCN in the method lead to a higher computational complexity of the algorithm? This should be discussed theoretically or empirically.

Author response:
In order to show the efficiency of the proposed AdapTable, we perform a computational efficiency comparison between test-time adaptation baselines and AdapTable in **Figure 6**. The averaged adaptation time is calculated by averaging adaptation time over all test instances in CMC dataset corrupted by Gaussian noise. We find that the adaptation time of AdapTable ranks third among eight TTA methods, by showcasing its computational tractability. Furthermore, we observe that our approach requires significantly less adaptation time compared to TTA baselines such as TTT++ [1], SAM [2], EATA [3], and SAR [4], despite constructing shift-aware graph and incorporating a single forward pass for GNN with negligible extra cost of adjusting output label estimation are required. This can be attributed to the fact that the graph we generate places each column as a node, resulting in a graph of a very small scale -- typically ranging from tens to hundreds of nodes. This minimizes the cost of message passing in GNN forward process, while other baselines iterate through multiple adaptation steps with forward and backward processes, leading to increased computational expenses. 

Furthermore, we also provide GNN post-training time of AdapTable under different scales in Table 5, encompassing small-scale (CMC, MLP), medium-scale (HELOC, FT-Transformer [5]), and large-scale (Diabetes Readmission, TabNet [6]). GNN post-training requires only a few seconds for small- and medium-scale settings, and notably, it remains negligible, even in our largest experimental setting.

We sincerely appreciate the reviewer again for the thoughtful comments.


[1] Yuejiang Liu, et al. "TTT++: When does self-supervised test-time training fail or thrive?" NeurIPS 2021.

[2] Pierre Foret, et al. "Sharpness-aware minimization for efficiently improving generalization" ICLR 2021.

[3] Niu, Wu, Zhang, et al. "Efficient test-time model adaptation without forgetting" ICML 2022.

[4] Niu, Wu, Zhang, et al. "Towards stable test-time adaptation in dynamic wild world" ICLR 2023.

[5] Yury Gorishniy, et al. "Revisiting deep learning models for tabular data" NeurIPS 2021.

[6] Sercan Ö Arik, et al. "TabNet: Attentive interpretable tabular learning" AAAI 2021.