Paper: CAST: Cluster-Aware Self-Training for Tabular Data (ICLR-2024)

Reviewer comment (Reproducibility):
The density estimation plays an important role in the proposed method. However, the authors only spend a few words saying that the density is estimated using the prior knowledge derived from the labeled training data distribution. I am confused when reading this part of the method and hope the authors can provide more details on that.

Author response:
We apologize for any confusion regarding the density estimation method in our manuscript. To clarify, we have revised the method section to detail the process as follows.

We get the estimated density for unlabeled samples by extracting the prior knowledge using a density estimator $\textit{D}_{t}$ (e.g. multivariate kernel density estimator or empirical likelihood) which is fitted to the labeled training data distribution
Here, the prior knowledge $\boldsymbol{\gamma}$ for each class is defined as follows:

$$
\boldsymbol{\gamma}^{(i)} \leftarrow \textit{D}_{t}(\boldsymbol{x}^{(i)}), \quad \text{where} \quad \boldsymbol{\gamma}^{(i)} = \text{[}{\gamma}_1, \gamma_2, ... , \gamma_N\text{]}
$$

Then, we normalize $\boldsymbol{\gamma}$ using a min-max scaler because the scale of $\boldsymbol{\gamma}$ varies among implementations, and we need a relative measure to align unlabeled samples.

And we leave the specific choice of density estimator for CAST to implementation as we believe open to extension is a crucial component for the versatility. For example, CAST which uses dataset-specific density estimator might be surpass CAST-D or CAST-L on the specific dataset. Our implementation of CAST-D and CAST-L are shown in Appendix D.

Reviewer comment (Experiments):
Just as the authors have claimed, the only difference between CAST and the conventional self-training algorithm is the use of regularized confidence. In other words, it seems that the proposed method has no specific designs for tabular data. Thus, I wonder if it is possible to supply a bit more results on other forms of data to show the proposed method is a general solution in self-training.

Author response:
We agree with your observation that the core idea of CAST, which involves adjusting the prediction confidence based on the cluster assumption, could extend to data types beyond tabular data. However, as we now note in our conclusion, the current implementation of CAST is limited by the absence of suitable density estimation methods for non-tabular data types, such as images or text. The structured nature of tabular data allows for clear density estimation from labeled training datasets as tabular data ensures that each feature occupies a specific, fixed position within the table. But there is no such characteristics in other data types. This limitation is now explicitly acknowledged in our manuscript.

Reviewer comment (Novelty):
There are some related self-training enhanced clustering methods such as SCAN (ECCV 2020), SPICE (TIP 2022), and TCL (IJCV 2022), that the authors are encouraged to include in the related works.

Author response:
After reading the suggested literature, we have included SPICE (TIP 2022) and TCL (IJCV 2022) in the related works section as follows: Niu et al. (2022) ensure the reliability of pseudo-labels through the use of a semantically consistent ratio, while Li et al. (2022) enhance clustering performance by selectively incorporating the most confident predictions from each cluster. We also agree that SCAN (ECCV 2020) is significant in semi- and self-supervised learning, but their work does not improve self-training. Therefore, we exclude it from our related works section as it does not align directly with our work on enhanced self-training.

Reviewer comment (Experiments):
The scope of applicability of the proposed solution is quite narrow. Indeed, the proposed solution seems to be applicable to low dimensional datasets and it is not clear how well the solution scales and generalises to more realistic high-dimensional datasets (**Significance**).

Author response:
*Answer:* We understand the concern about CAST's scalability to high-dimensional datasets. To address this, we included results from the Bioresponse dataset in OpenML-CC18, which is notably high-dimensional with 1777 features and 3751 samples. The results, detailed in Table 11 in Appendix I, demonstrate that CAST effectively handles high-dimensional data, showing significant improvements of CAST.

| | Method  | Relative Improvement | | Method  | Relative Improvement |
|---------|---------|----------|---------|---------|----------|
| | Baseline| -0.024    | | Baseline| 0.155   |
| | TS      | 0.052   | | TS      | 0.290  |
| | HB      | 0.022    | | HB      | 0.354    |
FPL| SP    | -0.058    |CPL| SP    | 0.265    |
| | GP      | 0.567    | | GP      | 0.249    |
| | CAST-D  | 0.688    | | CAST-D  | 2.454    |
| | CAST-L  | 1.150    | | CAST-L  | 1.821    |

Reviewer comment (Experiments):
The proposed solution requires a density estimation step and therefore it is more computationally demanding with respect to the considered baselines. Experiments should provide also this information (**Quality**).

Author response:
*Answer:* We have added detailed information about CAST’s computational cost in Appendix H. The computational costs for CAST-D and CAST-L were benchmarked against the training time of XGBoost, using a single CPU core of Ryzen 5975wx and one RTX 4090. The results indicate that CAST's computational demand is comparable to the training time of XGBoost and significantly lower than that of neural networks.

|            |            |            |           |         |         |    Time (s)  |       |         |          |           |          |            |
|------------|--------|--------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
|            |            | 6M mortality | |          | diabetes |       |          | ozone |           |           | cmc  |            |  
|            | XGB    | FT     | MLP   | XGB   | FT    | MLP   | XGB   | FT    | MLP   | XGB   | FT    | MLP   |
| Training   | 5.25   | 749.02 | 96.97 | 0.10  | 26.85 | 9.40  | 0.51  | 89.57 | 14.32 | 0.19  | 27.90 | 16.12 |
| CAST-D     |    | 7.13      |      |   | 0.15     |      |   | 0.50    |      |  | 0.25      |      |
| CAST-L     |  | 3.99      |     |  | 0.14      |      |   | 0.48     |      |   | 0.19     |     | 
|            |            |            |       Relative    |      additional   |     overhead    |       of CAST     |  compared     |     to training    |   time (%)        |           |          |            |
|            |            | 6M mortality | |          | diabetes |       |          | ozone |           |           | cmc  |            |  
|            | XGB    | FT     | MLP   | XGB   | FT    | MLP   | XGB   | FT    | MLP   | XGB   | FT    | MLP   |
| CAST-D   | 135.84 | 0.95   | 7.35  | 148.15| 0.57  | 1.63  | 98.72 | 0.56  | 3.51  | 127.70| 0.89  | 1.53  |
| CAST-L   | 76.08  | 0.53   | 4.12  | 133.65| 0.52  | 1.47  | 94.99 | 0.54  | 3.38  | 97.35 | 0.68  | 1.17  |

Reviewer comment (Experiments):
The cluster assumption (or equivalently the low-density separation) can be cheaply incorporated by leveraging techniques based on entropy minimisation for semi-supervised learning. A discussion and possibly experimental comparison against such techniques is missing (**Quality**). For instance, see [1-3].

Author response:
*Answer:* Self-training is a version of the entropy minimization algorithm, which minimizes the likelihood deprived of the entropy of the partition [1]. It constructs hard (one-hot) labels from high-confidence predictions on unlabeled data to implicitly achieve entropy minimization [2]. We have clarified that self-training is inherently an entropy minimization algorithm. And we acknowledge the significance of entropy minimization techniques in semi-supervised learning. However, we intentionally did not compare CAST with other SSL techniques including other entropy minimization algorithms as our aim is to demonstrate whether self-training can be improved by solely refining confidence or not. Note that, our manuscript’s central question is ‘Can we improve self-training for tabular data by making confidence more reliable, without altering the self-training algorithm or model architecture?’.

Reviewer comment (Writing):
The paper is not self-contained, i.e., there are gaps in the proposed methodology. For example, in Eq. 2 the authors state that prior knowledge is encoded in terms of a vector $\gamma$ and $\gamma$ is assigned the output of some function TD(). However, the function is not clearly described. Only "prior knowledge which is derived from the labeled training data distribution TD" is mentioned in the text. Probably TD is on purpose quite vague. However, there is not even a single mention of what this could be. It is also confusing that it seems to be the labeled training data distribution, however, this distribution should at maximum be implicitly given by the data sample.

Author response:
We apologize for any confusion regarding the density estimation process in our methodology. We have revised the methods section to provide a clearer explanation as follows.

We get the estimated density for unlabeled samples by extracting the prior knowledge using a density estimator $\textit{D}_{t}$ (e.g. multivariate kernel density estimator or empirical likelihood) which is fitted to the labeled training data distribution.
Here, the prior knowledge $\boldsymbol{\gamma}$ for each class is defined as follows:

$$
\boldsymbol{\gamma}^{(i)} \leftarrow \textit{D}_{t}(\boldsymbol{x}^{(i)}), \quad \text{where} \quad \boldsymbol{\gamma}^{(i)} = \text{[}{\gamma}_1, \gamma_2, ... , \gamma_N\text{]}
$$

Then, we normalize $\boldsymbol{\gamma}$ using a min-max scaler because the scale of $\boldsymbol{\gamma}$ varies among implementations, and we need a relative measure to align unlabeled samples.

And we leave the specific choice of density estimator for CAST to implementation as we believe open to extension is a crucial component for the versatility. For example, CAST which uses dataset-specific density estimator might be surpass CAST-D or CAST-L on the specific dataset. Our implementation of CAST-D and CAST-L are shown in Appendix D.

The reason for using the labeled training data distribution in CAST is to enable the regularization of confidence for each class. This is achieved by utilizing class-specific estimated density, which is derived from the labeled data distribution.

Reviewer comment (Reproducibility):
In Algorithm 1 the $\gamma$ does not even occur. I assume it is somewhere hidden in the $\Phi$ which supposedly does the pseudo-labeling. However, $\Phi$ is nowhere given concretely. Not even in the appendix — at least I could not find it there. Still, in the text, the authors write that Algorithm 1 is the complete algorithm but only a very basic self-training framework is given there — nothing special about CAST as a standalone method. Also, the loop is terminating with respect to some unknown termination condition of $\Phi$ which is neither elaborated.

Author response:
We appreciate your feedback on the lack of clarity regarding $\Phi$ in Algorithm 1. $\Phi$ is the pseudo-labeling algorithm that adopts equation (4), and its specific implementation can vary. CAST is designed to the following question that we raised in the Introduction section: ‘Can we improve self-training for tabular data by making confidence more reliable, without altering the self-training algorithm or model architecture?’. In essence, CAST is an enhanced self-training algorithm which regularizes the confidence of the classifier to be aware of the cluster assumption without altering the self-training algorithm. Therefore, we present the conventional self-training algorithm which adopts regularized confidence for pseudo-labeling as pseudo-code of CAST. And we did not specify a particular $\Phi$ within the pseudo-code, as CAST is designed to be compatible with any self-training algorithm, irrespective of the pseudo-labeler employed. Lastly, the termination condition for $\Phi$ is implementation-specific and hence was not explicitly defined.

Reviewer comment (Evaluation):
Speaking about the empirical evaluation: While relative improvements over a baseline might be the primary goal, with which I agree, it is relatively hard to interpret the significance of the results. In particular, it impedes the application of a statistical test whether the results are significant. From the results, the differences are probably significant but still it is very hard to interpret and I would prefer plain results even though metrics might differ.

Author response:
We understand the difficulty in interpreting relative improvements and have now included absolute performance in Table 1 and Table 11 in Appendix J. Additionally, we applied statistical tests to show that the performance improvements of CAST are not coincidence, whereas the calibrated confidence based self-trainings are. Concretely, there are some experiments that calibrated confidence based self-training outperform naive confidence based self-training, but statistical test verifies that calibrating the confidence is meaningless in the context of self-training.

Reviewer comment (Theory):
How is $\gamma$ computed? What are the requirements or desiderata for computing $\gamma$ to yield a sound approach?

Author response:
We get $\gamma$ using a density estimator $\textit{D}_{t}$ which is fitted to the labeled training data distribution. The natural characteristic of the tabular data is each feature occupies a specific, fixed position within the table. This allows us directly to extract prior knowledge from the labeled training dataset unlike other domains (e.g., image or text) where do not have such rules. There are various successful approaches regarding density estimation for tabular data [4,5]. We extract prior knowledge using multivariate kernel density estimator and empirical likelihood.

Reviewer comment (Reproducibility):
What is the termination criterion in relation to $\Phi$?

Author response:
The termination criterion depends on the implementation of $\Phi$. For example, if the $\Phi$ is fixed-threshold pseudo-labeling, the termination condition is when no performance gain is achieved. On the other hand, if the $\Phi$ is curriculum pseudo-labeling, it is when there are no unlabeled data left to label.

Reviewer comment (Reproducibility):
How is the pseudo-labeling $\Phi$ done?

Author response:
The pseudo-labeler $\Phi$ can be any algorithm suitable for pseudo-labeling, such as fixed-threshold pseudo-labeling or curriculum pseudo-labeling as we designed CAST to maintain versatility. It depends on the implementation of the user. For example, within fixed-threshold pseudo-labeling strategies, pseudo-labels are designated once their confidences meet or exceed a certain threshold. Meanwhile, curriculum pseudo-labeling strategies generate pseudo-labels based on a threshold but operate under the premise that samples with higher confidence are easier for the classifier to handle.

Reviewer comment (Writing):
From the Introduction, I cannot get the significant relationship between tabular data and the proposed self-training method. The motivation and organization of this paper should be further clarified.

Author response:
We acknowledge the need for a clearer exposition of the relationship between tabular data and our proposed self-training method. To address this, we clarify why we study the primary question, ‘Can we improve self-training for tabular data by making confidence more reliable, without altering the self-training algorithm or model architecture?’, in this paper for tabular data, specifically in the second paragraph of the Introduction as follows:

Various studies have proposed solutions to counteract the noise in pseudo-labels induced by erroneous confidence, but they have diminished the simplicity and versatility of self-training. Concretely, they often necessitate modifications to self-training algorithms or alterations in the model architectures (Li & Zhou, 2005; Tanha et al., 2017; Rizve et al., 2021; Seibold et al., 2022). Furthermore, most of them are not applicable to gradient boosting decision trees (GBDT) as they are designed for neural networks. These limitations pose a substantial impediment to practitioners who want to apply reliable self-training on the tabular data where GBDTs have been the dominant architectures (Kaggle, 2021; Borisov et al., 2022; Shwartz-Ziv & Armon, 2022). Therefore, we conclude that any enhanced self-training for the tabular domain must maintain simplicity and versatility. Consequently, we study a natural but ignored question: *Can we improve self-training for tabular data by making confidence more reliable, without altering the self-training algorithm or model architecture?*

Reviewer comment (Experiments):
In addition, the difficulties brought by tabular data over the general unstructured data (e.g., images, texts) in machine learning have not been discussed. In detail, they only stated that the GBDT is suitable for tabular data, while they do not explain why other methods are not suitable.

Author response:
We acknowledge the need for a clearer exposition of the relationship between tabular data and our proposed self-training method. To address this, we clarify why we study the primary question, ‘Can we improve self-training for tabular data by making confidence more reliable, without altering the self-training algorithm or model architecture?’, in this paper for tabular data, specifically in the second paragraph of the Introduction as follows:

Various studies have proposed solutions to counteract the noise in pseudo-labels induced by erroneous confidence, but they have diminished the simplicity and versatility of self-training. Concretely, they often necessitate modifications to self-training algorithms or alterations in the model architectures (Li & Zhou, 2005; Tanha et al., 2017; Rizve et al., 2021; Seibold et al., 2022). Furthermore, most of them are not applicable to gradient boosting decision trees (GBDT) as they are designed for neural networks. These limitations pose a substantial impediment to practitioners who want to apply reliable self-training on the tabular data where GBDTs have been the dominant architectures (Kaggle, 2021; Borisov et al., 2022; Shwartz-Ziv & Armon, 2022). Therefore, we conclude that any enhanced self-training for the tabular domain must maintain simplicity and versatility. Consequently, we study a natural but ignored question: *Can we improve self-training for tabular data by making confidence more reliable, without altering the self-training algorithm or model architecture?*

Reviewer comment (Experiments):
In my view, CAST is not a tabular data-specific method. Its idea is to adjust the prediction confidence based on the cluster assumption, which is also available for other data types. I think that this discussion should be included and the corresponding experiments are needed.

Author response:
We agree with your observation that the core idea of CAST, which involves adjusting the prediction confidence based on the cluster assumption, could extend to data types beyond tabular data. However, as we now note in our conclusion, the current implementation of CAST is limited by the absence of suitable density estimation methods for non-tabular data types, such as images or text. The structured nature of tabular data allows for clear density estimation from labeled training datasets as tabular data ensures that each feature occupies a specific, fixed position within the table. But there are no such characteristics in other data types. This limitation is now explicitly acknowledged in our manuscript.

Reviewer comment (Theory):
There lacks detailed analysis on why this method works. And I wonder whether it has some relationship with label smoothing techniques. And I would recommend the authors to give more in-depth analysis, either empirical or theoretical. Also, the 'cluster assumption' or the reliable pseudo-labels should be checked or verified in real datasets. For example, why they are reliable and can we explain it?

Author response:
We appreciate your insight on the need for a more comprehensive analysis of CAST's efficacy. To address this, we've included the following explanations in Appendix C. 

Self-training is a version of the entropy minimization algorithm, which minimizes the likelihood deprived of the entropy of the partition [1]. It constructs hard (one-hot) labels from high-confidence predictions on unlabeled data to implicitly achieve entropy minimization [2]. The entropy minimization techniques assume that the cluster assumption is ensured in the dataset [3], and aim that the classifier learns the low-density separations in the data. However, unreliable pseudo-labels that lie in low-density regions, stemming from erroneous confidence, violate the assumption and consequently disrupt the classifier's ability to learn the separations among classes. On the other hand, CAST forces the pseudo-labels in low-density regions to have lower confidence to avoid the violation of the assumption. Therefore, CAST achieves more reliable pseudo-labels resulting in successful entropy minimization.

And Label smoothing employs soft labels rather than hard labels, such as one-hot encoded vectors, with the aim of preventing the model from becoming overly confident in its predictions. In contrast, CAST specifically aims to reduce the confidence of pseudo-labels in low-density regions, ensuring that unreliable pseudo-labels are not generated. The labels used in CAST remain hard labels. 

Lastly, we demonstrate that pseudo-labels in high-density regions are more accurate than those in low-density regions in Figure 1 using 6-month mortality prediction post-acute myocardial infarction (6M mortality) dataset which is sourced from Korea Acute Myocardial Infarction Registry (KAMIR). The Figure 1 shows the heuristic evidence supporting for the cluster assumption.

Reviewer comment (Experiments):
I believe there are many many semi-supervised learning methods, but there are only 5 baselines, which I think is not enough and representative for SSL.

Author response:
We acknowledge the extensive variety of semi-supervised learning (SSL) methods available such as VIME [4]. However, most of the people have used GBDTs for tabular data which are not compatible with most of SSL approaches. Therefore, we focus on self-training which is universal applicable approach and study ‘Can we improve self-training for tabular data by making confidence more reliable, without altering the self-training algorithm or model architecture?’. Hence, our research intentionally does not compare CAST with other SSL techniques because our objective is not to challenge the efficacy of these techniques per se, but to demonstrate that self-training can indeed be improved through refined confidence without modifying the self-training algorithms or model architectures.

Reviewer comment (Reproducibility):
Some of the datasets are not open-sources, e.g., 6M mortality.

Author response:
We understand the concern regarding the use of the non-open source 6M mortality dataset. To balance this, we incorporated 19 open-source datasets from OpenML-CC18, providing a broad and diverse evaluation of CAST. These datasets, coupled with the 6M mortality dataset, offer a robust assessment of CAST’s performance and versatility. The consistent results across these varied datasets confirm CAST's effectiveness and its applicability to a wide range of scenarios.