Paper: Data Imputation by Pursuing Better Classification: A Supervised Learning Approach (ICLR-2024)

Reviewer comment (Reproducibility):
Some key points (in particular, how parameters can be set) are not addressed.

Author response:
**R4.1 The settings and the choice of the model parameters.**

Thank you for pointing out this issue in detail. It is indeed a meaningful consideration for users. We empirically offer some suggestions regarding the parameter settings. Regarding the adjustment of the parameter $\gamma$, we adopt the traditional "grid search" approach commonly used in SVM to select the parameter value.  For $\eta$, which controls the extent to which the similarity between samples can be modified, if there is still room for improvement in the predictive accuracy of the training set, reducing $\eta$ can be explored to seek better similarity relationships among samples. As for $r$, which governs the robustness of the classifier, if the predictive accuracy during training is already high but the performance on the holdout set is unsatisfactory, increasing the value of $r$ can help alleviate this issue. We wish that the above response partially addresses your concerns.

Reviewer comment (Evaluation):
The results only mildly support the claim that the approach is superior to the others: they are often (but not always) better, but by a small margin only (and the difference cannot be deemed significant), and they are reported over four datasets only.

Author response:
**R4.2 Studying on more data sets.**

We sincerely appreciate your valuable feedback. Following your suggestion, we conducted experiments on two new data sets. For detailed results of the experiments, please refer to **R0.1** in the Author Rebuttal.

Reviewer comment (Novelty):
The presentation of existing works is somehow a bit short: many works on learning from missing data or classification from missing data were not mentioned. There is no discussion on the nature of the missingness process here. The data seem to be considered as missing at random.

Author response:
**R4.3 Studying on other works and other missingness processes.**

We sincerely appreciate your insightful comments. Based on your suggestions, we have included a comparison of results on two data sets with inherent missing values. And we compare the results with a recent neural network-based method, NeuMiss [1]. For more details, please refer to **R0.1** in the Author Rebuttal.

Reviewer comment (Experiments):
The remark regarding the results obtained for "extreme" values of $\gamma$ seem somehow obvious: the cases covered either correspond to instances being all dissimilar ($\gamma=1/32$), or similar ($\gamma=32$), hence the results. This raises the question of the sensitivity of your approach to the choice of $\gamma$—or, more generally, to the model parameters, the choice of which appears to be crucial, and actually very difficult to make without strong assumptions. Could you elaborate on that?

Author response:
**R4.1 The settings and the choice of the model parameters.**

Thank you for pointing out this issue in detail. It is indeed a meaningful consideration for users. We empirically offer some suggestions regarding the parameter settings. Regarding the adjustment of the parameter $\gamma$, we adopt the traditional "grid search" approach commonly used in SVM to select the parameter value.  For $\eta$, which controls the extent to which the similarity between samples can be modified, if there is still room for improvement in the predictive accuracy of the training set, reducing $\eta$ can be explored to seek better similarity relationships among samples. As for $r$, which governs the robustness of the classifier, if the predictive accuracy during training is already high but the performance on the holdout set is unsatisfactory, increasing the value of $r$ can help alleviate this issue. We wish that the above response partially addresses your concerns.

Reviewer comment (Experiments):
It is not clearly stated whether the missing values are in the training or test data (or both).

Author response:
**R4.6 Missing values: Presence in training/test sets or not.**

In our experimental setup, the training set contains missing values while the test set does not, which follow the settings in [4].

Reviewer comment (Theory):
Does ignoring the PSD constraint to solve the problem and then projecting back onto the space of PSD matrices have an impact on the result obtained?

Author response:
**R4.7 The influence of dealing with the PSD constraint.**

Such an operation does not affect the results and is a widely applicable operation. For detailed theoretical references, please refer to the Theorem 2.1 in [5] and the Equation (9) in [6].

Reviewer comment (Theory):
Should Step 2 be skipped in the first phase, what would be the outcome of the proposed strategy? It seems that this amounts to implicitly assume that the data are "perfect": can you provide any insights regarding this?

Author response:
**R4.8 The outcome when skipping the Step 2 in the first phase.**

If we skip Step 2, the flexibility of our framework (allowing changes in both data similarity and classifier) makes it susceptible to overfitting. In such a scenario, if the allowed range for $K_\Delta$ to change is large (i.e., when the value of $\eta$ is set too small), it can lead to the situation you mentioned as "perfect data"，where the resulting kernel matrix tends to approach the ideal kernel matrix $yy^T$ as closely as possible. Although it results in high classification accuracy on the training set, it may not generalize well to the test set.

Reviewer comment (Theory):
I do not understand why $\varepsilon^*$ should be positive definite (phase I, step 2, Equation (9), page 6). For complete consistency with Stage I, shouldn't $\varepsilon$ (or $\varepsilon^*$) be used as well in Stage II?

Author response:
**R4.9 Discussions about the $\mathcal{E}^\ast$.**

a) We set $\mathcal{E}^\ast$ as positive semi-definite (PSD) to ensure that the kernel matrix used in SVM remains PSD, as supported by the Schur Product Theorem [7].

b) It is important to note that that we pursue the similarity relationship between data with better classification performance by optimizing $K_\Delta$, and use these relationships to guide the data imputation process. The introduction of $\mathcal{E}^\ast$ is solely intended to enhance the robustness of the classifier. Therefore, in Stage II, where we focus on data feature imputation, we only utilize $K_o\odot K_\Delta$.

Reviewer comment (Writing):
Page 1, "we typically use subsets of indices [...] with $\bfseries{x}_{\bfseries{o}_i}^i$": clearly define these notations.

Author response:
**R4.10 The modification on writing.**

a) Page 1. we typically use the set of observable features $o_i\subseteq 2^{\lbrace 1,\cdots,d\rbrace }$ and missing features $\ast$ to represent the missing data $x_{o_i}^i\in(\mathbb{R}\cup \{\ast\})^d$. This sequence is then consolidated into an incomplete data set $\mathcal{D}=\lbrace x_{o_i}^i, y_i\rbrace _{i=1}^N$.

Reviewer comment (Writing):
Page 1, "the importance of labels has not been fully taken into account": this statement, somehow a bit assertive, is difficult to understand.

Author response:
**R4.10 The modification on writing.**

b) Page 1. However, the effective utilization of label information has not been fully explored yet.

Reviewer comment (Writing):
Page 4, Section 3.1, paragraph "Notations": sentences should not begin with a mathematical symbol.

Author response:
**R4.10 The modification on writing.**

d) Page 4. The set of real numbers is written as $\mathbb{R}$. The set of integers from 1 to $N$ is written as $[N]$. The cardinality of the set $\mathcal{A}$ is written as $|\mathcal{A}|$ denotes. We take $a$, $\boldsymbol{a}$, and $\boldsymbol{A}$ to be a scalar, a vector, and a matrix, respectively. Let $\boldsymbol{0}$ and $\boldsymbol{1}$ denote vectors consisting of all zeros and all ones with the appropriate size. The inner product of two vectors in the given space is written as $\langle\cdot, \cdot\rangle$. We take $\mathrm{diag}(\boldsymbol{a})$ to be an operator that extends a vector to a diagonal matrix. The Frobenius norm of a matrix is written as $\lVert \cdot \rVert_\mathrm{F}$. The set of positive semi-definite (PSD) matrices is written as $\mathcal{S}_{+}$. The Hadamard product is written as $\odot$.

Reviewer comment (Writing):
Page 4, Section 3.2: matrix $\mathbf{K}_\Delta$ is not properly introduced.

Author response:
**R4.10 The modification on writing.**

e) Page 4. Initially, we compute the observed kernel matrix $K_o$, and then optimize an adjustment matrix $K_{\Delta}$ with the same dimension to derive $\tilde{K}=K_o\odot K_{\Delta}$ as the imputed kernel matrix.

Reviewer comment (Writing):
Page 5, "is a semi-definite programming": should be "is a semi-definite program".

Author response:
**R4.10 The modification on writing.**

f) Page 5. The problem above is a semi-definite program, which is generally computationally expensive to solve.

Reviewer comment (Writing):
Page 6, $m$ does not seem to be properly introduced.

Author response:
**R4.10 The modification on writing.**

g) Page 6. The essence of this task involves solving a nonlinear system of equations. Additionally, by defining the missing ratio of data features as $m$, the aforementioned system consists of $N(N-1)/2$ equations and $Ndm$ variables.

Reviewer comment (Writing):
Page 7, "including \textit{australian}, [...]": should be "namely \textit{australian}, [...]".

Author response:
**R4.10 The modification on writing.**

h) Page 7. We chose four real-world data sets from libsvm (Chang & Lin, 2011), namely australian, german, heart and pima (a.k.a. diabetes).

Reviewer comment (Experiments):
There is a long history of research on imputation of data missing at random. Thus, comparing to only 4 other baselines on 4 small data sets (both in number of examples and features) might not be comprehensive enough. It is not clear why those particular 4 data sets were selected (other than being very small).

Author response:
We sincerely appreciate your valuable feedback. The additional comparisons you mentioned regarding the datasets and imputation methods are indeed important for validating the performance of our algorithm, and other reviewers have also expressed similar concerns. Therefore, following your suggestion, we conducted experiments using the latest neural network-based method, NeiMiss [1], on two additional datasets. And we have provided detailed results in **R0.1** in the Author Rebuttal. We hope this addresses some of your concerns.

Reviewer comment (Experiments):
It would be useful to show the computational cost for the performed experiments. The largest data set used in the experiments has only 1000 examples. Is this because the proposed method is too expensive to run on larger data?

Author response:
Regarding the issue of scalability to large-scale data in our proposed method, it is indeed a challenge due to the need for leveraging pairwise information in the samples. Additionally, some popular techniques used to accelerate kernel methods cannot currently be applied to the imputation problem. This issue will also be a research direction that we will dedicate efforts to in the future. For a more detailed discussion, please refer to **R0.2** in the Author Rebuttal. If you have any specific questions, we are more than happy to engage in further discussions with you over the next week.

Reviewer comment (Experiments):
My main concern is related to the experiments, that have several issues, in my opinion. Studying only 4 small data sets ($N\leq 1,000$) is not particularly compelling, especially given that most standard deviations are quite important (in Table 3, I doubt the author's technique is statistically significantly better than mean imputation in most scenarios).

Author response:
**R1.1 Studing on more and larger data sets.**

We sincerely appreciate your valuable feedback. Following your suggestion, we conducted experiments on two new data set. For detailed results of the experiments, please refer to **R0.1** in the Author Rebuttal. As for the lack of involvement with large-scale data sets, we provided a comprehensive discussion in **R0.2** in the Author Rebuttal.

Reviewer comment (Evaluation):
Mean imputation appears to be, by far, the best method (if we ignore the author's method). This is not very consistent with the literature. For instance, in the genRBF paper (Smieja et al., 2019), genRBF is on par or better than the mean on the "Australian" data set, while it is much worse than the mean in this submission. Similarly, in the GEOM paper (Chechik et al., 2008), GEOM is essentially always on par with the mean, and is generally worse in this submission.

Author response:
**R1.2 New experimental results for two tested items in Australian and one tested item in German.**

Your questions about the accuracy reported for MI, genRBF, and GEOM may be related to our experimental setup. For specific examples, we will discuss them as follows.

a) For MI, as described in [2], it allows leveraging feature information from the test set to aid in imputation. In our experimental settings, we only introduced missing features in the training set while utilizing complete holdout and test sets. Therefore, especially in cases of high feature missing rates, the information from the test set contributes to better classification performance for MI. 

b) For genRBF, some abnormal results in extreme cases were due to the narrow range used to adjust hyperparameters. We have modified the grid search range for $C$ and $\gamma$ and made our best efforts to improve the performance, and then we have adjusted the values of genRBF for several tested items. For the case of Australian $(m=60\%)$, Australian $(m=80\%)$, and German $(m=80\%)$, based on our correction, genRBF should have the accuracy of $0.796 \pm 0.041$, $0.704 \pm 0.058$, and $0.687 \pm 0.045$, respectively. And for other scenarios, we carefully examined and confirmed that the previously used parameters were optimal. Thank you for meticulously pointing out this issue and we will revise these results in our manuscript.

c) For GEOM, we conducted experiments specifically for the data-missing scenario of $m=90\%$ (i.e., the experimental setup described in their paper), using the provided code by the authors. On the Pima dataset, we obtained a result of $0.658 \pm 0.029$ for the GEOM method, which aligns with the reported result of $0.66 \pm 0.05$ in the Table 1 of [2]. However, we obtained a result of $0.689 \pm 0.036$ for the MI, which is higher than the reported value of $0.65 \pm 0.04$ in the paper. This discrepancy arises from the utilization of the complete test set, as we mentioned above.

Reviewer comment (Theory):
The authors do not study the theoretical properties of their methods. In particular, the assumptions on the missingness mechanism are not discussed. All experiments use missing completely at random (MCAR) data, and the authors do not discuss this experimental design choice. Studying (empirically and/or theoretically) whether or not this algorithm works on non-MCAR data would be interesting.

Author response:
**R1.3: Studing on the non-MCAR data.**

We sincerely appreciate your valuable feedback. Based on your suggestions, we have included a comparison of results on two data sets with inherent missing values, which are non-MCAR. For more details, please refer to **R0.1** in the Author Rebuttal.

Reviewer comment (Theory):
The paper reads generally well, but the mathematics are at times quite unclear. Several objects are not properly defined and some facts are not really proven. For instance, I imagine $K_0 = \exp ( - \gamma \sum_{p \in o_i \cap o_j} (x^i_p - x^j_p)^2 )$, but $K_0$ is never defined.

Author response:
**R1.4: Modifications in object definitions and the mathematics.**

We sincerely appreciate your meticulousness in pointing out our oversights.

a) Regarding the definition of $K_o$, as you correctly noted, it should indeed be defined by the first term in the last line of Equation (3).

b) In regards to the symbol $\ast$, it represents an unknown real number in future manuscript, which aids in computing the range of values such as $x_p^i-\ast$, assuming that $x$ is normalized to the range [0,1].

c) Thank you very much for your detailed observation. Indeed, our wording was not precise. Clipping a matrix first and then projecting it into the PSD matrix space does not guarantee that every element of the resulting matrix remains within the proper range. Our intention in introducing the proper range was merely to prevent significant changes in $K_\Delta$. While there may be elements slightly exceeding the range after the projection, these modified values can still be bounded, which allows the original constraint to remain effective and leads to similar performance in subsequent algorithms.

We will make the necessary modifications and additions in the manuscript to address the issues you have pointed out in the manuscript.

Reviewer comment (Theory):
It is unclear why $K^*_\Delta$ in Equation (6) is in the proper range (between $B_l$ and $B_u$). After clipping, you project on positive semidefinite matrices—why is it guaranteed that it would still be in the right range?

Author response:
**R1.4: Modifications in object definitions and the mathematics.**

We sincerely appreciate your meticulousness in pointing out our oversights.

a) Regarding the definition of $K_o$, as you correctly noted, it should indeed be defined by the first term in the last line of Equation (3).

b) In regards to the symbol $\ast$, it represents an unknown real number in future manuscript, which aids in computing the range of values such as $x_p^i-\ast$, assuming that $x$ is normalized to the range [0,1].

c) Thank you very much for your detailed observation. Indeed, our wording was not precise. Clipping a matrix first and then projecting it into the PSD matrix space does not guarantee that every element of the resulting matrix remains within the proper range. Our intention in introducing the proper range was merely to prevent significant changes in $K_\Delta$. While there may be elements slightly exceeding the range after the projection, these modified values can still be bounded, which allows the original constraint to remain effective and leads to similar performance in subsequent algorithms.

We will make the necessary modifications and additions in the manuscript to address the issues you have pointed out in the manuscript.

Reviewer comment (Writing):
I find it a bit odd to call $\mathcal{E}$ a "noise", since it is not something random but something that you optimize.

Author response:
**R1.5 Discussions about the ‘minor things’.**

a) Thank you for your suggestion. We will revise it to "perturbation matrix" in the subsequent manuscript.

Reviewer comment (Writing):
I also find the phrase "imputed kernel matrix", used a few times (in different forms) a bit odd: this matrix is always a complete matrix; it is the dataset used to build it that needs to be imputed.

Author response:
**R1.5 Discussions about the ‘minor things’.**

b) Regarding the phrase "imputed kernel matrix", we will replace it with "optimized kernel matrix" in order to express the intended meaning more clearly.

Reviewer comment (Novelty):
There has been a significant amount of work on supervised learning with missing values recently. Some of these papers could be interesting to discuss, for instance:  
Josse et al., On the consistency of supervised learning with missing values, arXiv:1902.06931, 2020  
Le Morvan et al., What’s a good imputation to predict with missing values? NeurIPS 2021  
Bertsimas et al., Beyond Impute-Then-Regress: Adapting Prediction to Missing Data, arXiv:2104.03158, 2022

Author response:
**R1.5 Discussions about the ‘minor things’.**

c) Thank you very much for providing these articles on handling missing values in supervised learning. We have included one of the methods, NeuMiss [3], in **R0.1** and **R0.2** for comparison with our approach. The mentioned articles are indeed very interesting, and we will incorporate and discuss these works in our future manuscript to further enhance its quality.