Paper: Improving Generalization for Missing Data Imputation via Dual Corruption Denoising Autoencoders (ICLR-2024)

Reviewer comment (Novelty):
The proposed method does not look new and either injecting noises or randomly masking some input features could serve as regularizers and improve the generalizability of neural network models, which is a well-known thing. Thus, I do not see too much novelty in the current proposed method.

Author response:
You raise a fair concern about using noise and masking as regularizers. As you noted, our key contributions are comprehensively studying the concurrent application and interplay of both techniques for missing data imputation. To our knowledge, this is the first work combining masking and noise injection for imputation while analyzing their complementary effects. The balanced loss function further governs their interaction. While building on prior techniques, we believe this tailored amalgamation provides unique insights.

Reviewer comment (Experiments):
The method is only evaluated on 5 datasets whose feature numbers are very small, which may not be sufficient to reflect the dataset we typically encounter in real-world applications. So I would suggest the authors add more experiments on datasets with large dimensions.

Author response:
Excellent point on evaluating more complex data. As future work, we will expand our experiments to high-dimensional tabular, image, text and time series datasets.

Reviewer comment (Evaluation):
Still about the evaluation pipeline, the performance did not outperform other methods in a lot of testing cases.

Author response:
You rightly note that our approach does not outperform on all datasets and missingness levels. As you noted, MIWAE, HI-VAE and MIDA all use overcomplete hidden spaces, giving them greater model capacity. This likely explains their superior performance on Glass and Spam datasets, as we demonstrate in Section 5.6.3 that adopting overcomplete representations in DC-DAE can further improve its accuracy.

Reviewer comment (Evaluation):
Another question or one evaluation I would like to see: even if there are differences in the "filled-in" missing values across different methods, does the lower NRMSE necessarily indicate a higher downstream-task performance? I mean, for example, what will happen if we apply a downstream task (say, a classification task) to these imputed datasets, how much performance difference we would expect in the classification accuracy?

Author response:
Analyzing downstream task accuracy is an excellent idea. We will train classifiers on the imputed data and report differences in classification performance in the future work. This will provide insights into real-world utility.

Reviewer comment (Novelty):
The authors should also mention other imputation methods (like methods using traditional machine learning) in Related Work besides the GAN-based and VAE-based ones.

Author response:
We agree on the importance of discussing traditional ML imputation techniques. However, our work is primarily focused on deep-learning based methods. As you noted, due to paper length constraints, we opted to briefly mention traditional techniques in the introduction without detailed coverage in the related work section. We can clarify this decision to focus the paper scope on deep learning techniques while acknowledging the broader imputation literature.

Reviewer comment (Theory):
Based on the loss function and the framework Fig.1, it looks like Formula (5) is not correct.

Author response:
Thank you for catching the Equation 5 error - we will update Figure 1 for conformity.

Reviewer comment (Novelty):
The technique novelty is limited. The benefits of this method are mainly for empirical/experimental observation. The dual corruption is to combine two ways of data augmentation during training, which is not technically innovative.

Author response:
You raise a fair point about novelty of the dual corruption technique. Our key contributions are comprehensively analyzing the concurrent application of both techniques for missing data imputation, when previous DAE-based methods had focused on one corruption or the other without justification. To the best of our knowledge, this is the first work that combines masking and noise injection for imputation and provides an extensive study quantifying their complementary effects. Furthermore, we propose a balanced loss function to delicately control the tradeoff between reconstructing artificial missingness versus denoising observed values. This governs the interplay between the dual corruptions. While building upon existing techniques, we believe this novel amalgamation tailored for imputation provides unique insights and capabilities.

Reviewer comment (Experiments):
The quantitative analysis of balanced loss function is not sufficient. Are there any specific experiments to demonstrate the Imputation performance varied with the tuning of the loss function?

Author response:
Excellent suggestion to analyze loss tuning. As Figure 2c demonstrates, the alpha hyperparameter controlling the rebalanced loss prevents overfitting and improves accuracy.

Reviewer comment (Evaluation):
More metrics are needed for the performance evaluation. In this work, there is only NRMSE as the imputation performance evaluation metric, are there any metrics for real-world applications, for example, classification accuracy? In one closely related work, DAEMA, the authors also use classification accuracy as one metric in addition to the NRMSE.

Author response:
Evaluating downstream performance is an important direction we will explore in future work, to demonstrate real-world applicability.

Reviewer comment (Experiments):
In the Table2, there is one result of Imputation performance comparison, I notice that the “Glass” result of DAEMA is different (higher value is reported) from the DAEMA paper (Table 1). I wonder if you have used the same parameters in DAEMA for fair comparison?

Author response:
Thank you for catching the DAEMA discrepancy. As you noted, the original DAEMA has far greater capacity. For fair comparison, we shrank the hidden space as described in Section 5.4. This highlights the capability of DC-DAE despite the compact architecture.

Reviewer comment (Evaluation):
Are there any quantitative results (NRMSE) from the balanced loss function?

Author response:
Excellent suggestion to analyze loss tuning. As Figure 2c demonstrates, the alpha hyperparameter controlling the rebalanced loss prevents overfitting and improves accuracy.

Reviewer comment (Evaluation):
The findings presented in the article exhibit a modest degree of robustness. In Table 2, it is evident that the proposed algorithm outperforms existing methods in three out of the five datasets. However, the discernible effectiveness of DC-DAE remains somewhat obscured.

Author response:
Firstly, you make an excellent point regarding DC-DAE's performance at high missing rates. While our method achieves top accuracy on 3 out of 5 datasets, as you noted, the overcomplete representations used by MIWAE, HI-VAE and MIDA likely explain their superior results for Glass and Spam. As shown in Section 5.6.3, adopting overcomplete hidden spaces in DC-DAE could further improve its performance. We will update our claims accordingly and acknowledge the impact of model capacity.

Reviewer comment (Evaluation):
When considering a missing rate of beta 0.8, the average NRMSE for DC-DAE does not rank highest; MIWAE yields superior results. Consequently, the assertion that DC-DAE significantly surpasses other methods may require further substantiation, as its advantage may be marginal in magnitude.

Author response:
Firstly, you make an excellent point regarding DC-DAE's performance at high missing rates. While our method achieves top accuracy on 3 out of 5 datasets, as you noted, the overcomplete representations used by MIWAE, HI-VAE and MIDA likely explain their superior results for Glass and Spam. As shown in Section 5.6.3, adopting overcomplete hidden spaces in DC-DAE could further improve its performance. We will update our claims accordingly and acknowledge the impact of model capacity.

Reviewer comment (Evaluation):
Upon revisiting the DC-DAE algorithm, it becomes apparent that hyperparameters have been introduced to modulate the loss function governing the interplay between a conventional Denoising Autoencoder (DAE) and a DAE operating with a noisy mask. The loss function comprises two distinct components, with hyperparameters acting as adjustable weights that govern the balance between these components. The regularization introduced by this weighting mechanism appears to lack substantial potency and exhibits limited efficacy.

Author response:
Secondly, I appreciate you confirming that our description of using the alpha hyperparameter to balance the loss function is correct. As Figure 2c demonstrates, this reweighting prevents overfitting to either the artificial missingness or noised observed data, contributing to the overall accuracy gains.

Reviewer comment (Writing):
The paper should provide clearer explanations of mathematical notations. For example, when using the "*" operation to denote element-wise multiplication, it would be beneficial to explicitly mention that this operation refers to the Hadamard product. Additionally, consider adopting standard tensor notation for better precision in mathematical representation.

Author response:
Finally, thank you for the notation suggestion - we will revise the methodology section to make the mathematical representations more precise and concise.

Reviewer comment (Writing):
In Section 4.4, the paper can use a more concise notation, explicitly stating the use of the L2 norm for the loss function. This will help the explanation of the loss term and enhance reader comprehension.

Author response:
Finally, thank you for the notation suggestion - we will revise the methodology section to make the mathematical representations more precise and concise.

Reviewer comment (Novelty):
The application of dual corruption techniques could be viewed as an incremental advancement rather than a breakthrough innovation, given the widespread use of such techniques in the field of deep learning.

Author response:
You raise a fair point that masking and noise injection have been explored individually in prior work. Our key contributions are comprehensively analyzing the concurrent application of both techniques for missing data imputation, when previous DAE-based methods had focused on one corruption or the other without justification. To the best of our knowledge, this is the first work that combines masking and noise injection for imputation and provides an extensive study quantifying their complementary effects. Furthermore, we propose a balanced loss function to delicately control the tradeoff between reconstructing artificial missingness versus denoising observed values. This governs the interplay between the dual corruptions. While building upon existing techniques, we believe this novel amalgamation tailored for imputation provides unique insights and capabilities.

Reviewer comment (Evaluation):
There is an absence of statistical details such as standard deviations in the reported results, particularly in Table 2. This omission raises questions regarding the statistical significance of the observed improvements in normalized root-mean-square error (NRMSE) when using the proposed method.

Author response:
Excellent point. We will report standard deviations for the results in Table 2 to provide insights into significance.

Reviewer comment (Experiments):
The paper focuses on tabular data and does not extend the evaluation to more complex or large-scale datasets where deep learning methods typically excel, potentially limiting the generalizability of the findings.

Author response:
You raise a fair concern regarding evaluation on more complex datasets. We focused on tabular data due to its prevalence in missing data problems. As future work, we will explore applying DC-DAE to image, text, and time series datasets with missingness.

Reviewer comment (Presentation):
The manuscript exhibits inconsistencies in citation formatting (notably [6] in Section 4.3). Could the authors ensure uniformity in referencing throughout the document to adhere to the submission guidelines?

Author response:
Thank you for noting the inconsistent citation formatting. To adhere to submission guidelines, we will unify the formatting style throughout the paper.

Reviewer comment (Writing):
Regarding equation (4), the manuscript suggests noise is added to the observed values only. Could the authors confirm whether the noise is exclusively added to these values and, if so, clarify the notation to reflect this selective application as `e * (1-M)`?

Author response:
You correctly point out that noise is only added to observed values in Equation 4. We will update this notation to e * (1-M) as you suggest, clearly indicating noise is only applied to non-missing entries.

Reviewer comment (Presentation):
In Figure 1, the relationship between the Mask Matrix and the Computed Data matrix is not depicted. Could the authors elaborate on how the Mask Matrix is incorporated into the model's computations, particularly in relation to equations (4)-(6)?

Author response:
We will modify Figure 1 to explicitly show how the mask matrix is element-wise multiplied with the data matrix to generate the corrupted input fed into the model.

Reviewer comment (Theory):
In equation (6), the DAE includes a mask matrix as part of its input. Could the authors clarify if this is a standard approach for DAEs and if the artificial mask matrix (M bar) is intended to help the model differentiate between real and artificially introduced missing data?

Author response:
Good point – providing the mask matrix as input is common practice in missing data imputation to help the model impute the missingness, although less standard in regular DAE. We will clarify this motivation and accordingly update Equation 6, where the corrupted mask matrix is denoted as M * M_bar.

Reviewer comment (Experiments):
Given the introduction of synthetic missing values during training, the manuscript should address how the model accounts for the potential distribution shift compared to the test data which lacks these artificial missing values.

Author response:
To mitigate training-test distribution shift, we ensure the model architecture and training process have no way to discriminate between artificial versus natural missing data. With both forms of missingness introduced randomly, there is theoretically no systemic difference in their distributions. Thus the model learns to impute all missing data during training.

Reviewer comment (Experiments):
When Missing Not At Random (MNAR) data is addressed, especially as outlined in Appendix A.1, how does the model manage instances where selected features for artificial missingness are also naturally missing? Does the model incorporate a mechanism to prevent learning to discriminate between real and synthetic missing values?

Author response:
Excellent observation – As you note, overlapping artificial and natural missingness is possible. We will clearly state the model takes their union as the total missingness. By preventing the model from distinguishing the two forms, it learns to impute any and all missing values.

Reviewer comment (Evaluation):
Considering the moderate size of the datasets utilized, could the authors justify their choice of validation method? Would multi-fold cross-validation not be a more suitable approach to assess model performance robustly?

Author response:
You raise a valid concern about our validation methodology. To more rigorously evaluate performance, we will incorporate n-fold cross-validation rather than relying solely on random seed trials given the dataset sizes. We will incorporate this improvement in the future work.

Reviewer comment (Evaluation):
The manuscript briefly mentions the utilization of overcomplete representations in Section 5.6.3. Could the authors discuss the trade-offs involved with this approach, particularly concerning model complexity and the potential for overfitting?

Author response:
As Figure 4 shows, overcomplete representations can further improve DC-DAE accuracy when missing rates are low or sample sizes large, but can decrease performance under high missingness and small dataset due to overfitting, as you observed.

Reviewer comment (Evaluation):
In scenarios where GAN components provide incremental gains as mentioned in Section 5.6.2, could the authors provide further insights into the conditions that favor their inclusion and offer recommendations on optimizing these components for consistent improvements?

Author response:
We agree that given the limited and inconsistent gains from GAN components in Figure 3b, providing integration guidelines is out of scope for this work focused on dual corruptions.

Reviewer comment (Novelty):
The originality of this paper, which proposes a method for data imputation using dual corruption strategies, may not be seen as highly innovative. The techniques of masking and adding noise have been well-explored in deep learning research as data augmentation techniques.

Author response:
Regarding originality, you raise a fair point that masking and noise injection have been explored in other contexts like data augmentation. Our key innovations are 1) concurrently applying both noise and masking corruptions specifically for missing data imputation, and 2) using a balanced loss to control the tradeoff between reconstructing artificial missingness vs observed values. We agree that we should better highlight these novel aspects.

Reviewer comment (Novelty):
Although the paper combines masking and adding noise with a balanced loss function, it does not verify the originality of this paper.

Author response:
Regarding originality, you raise a fair point that masking and noise injection have been explored in other contexts like data augmentation. Our key innovations are 1) concurrently applying both noise and masking corruptions specifically for missing data imputation, and 2) using a balanced loss to control the tradeoff between reconstructing artificial missingness vs observed values. We agree that we should better highlight these novel aspects.

Reviewer comment (Novelty):
It would be great to clarify how the proposed method differentiates from data augmentation techniques.

Author response:
Compared to standard data augmentation, our key differences are: 1) We introduce missingness rather than just augmenting existing data, which is critical for imputation. 2) The loss rebalancing prevents overfitting to the artificial missingness distribution. We will add this explanation of how our approach is tailored for missing data problems.

Reviewer comment (Writing):
On page 4, the notation for noise 'e' should be distinguished from regular text using a special character or formatting.

Author response:
We will distinguish the noise notation 'e' properly as suggested.

Reviewer comment (Experiments):
Section 5.6.1 needs clearer explanations. It should clearly explain why different sample sizes are used and what the numbers in the table mean. The figure shows many numbers, but the explanation doesn't analyze it comprehensively and quantitatively.

Author response:
For Section 5.6.1, we will expand the explanation of the results. The different sample sizes show ablation experiments on subsets of the full dataset to analyze trends. And the tables report performance relative to the baseline for that setting, with lower percentages indicating improved accuracy from our proposed techniques. We will add more details on the aims and insights from this ablation study.

Reviewer comment (Presentation):
For clarity and presentation, center Figures 3 and 4 in the document. Additionally, the numerical data within Figures 2, 3, and 4 need to be larger for easier readability.

Author response:
Thank you for the formatting suggestions. We will center Figures 3-4 and enlarge the data in Figures 2-4 to improve clarity.

Reviewer comment (Evaluation):
The analysis of Table 2 could be more detailed. For example, why does DC-VAE not perform well in the Glass and Spam dataset?

Author response:
For Table 2, you raise a good point. The relatively weaker Glass and Spam performance seems to be DC-DAE's compact architecture. As you noted, MIWAE, HI-VAE and MIDA all use overcomplete hidden spaces, giving them greater model capacity. This likely explains their superior performance on Glass and Spam datasets, as we demonstrate in Section 5.6.3 that adopting overcomplete representations in DC-DAE can further improve its accuracy.

Reviewer comment (Novelty):
What is the difference between the proposed method and data augmentation?

Author response:
Compared to standard data augmentation, our key differences are: 1) We introduce missingness rather than just augmenting existing data, which is critical for imputation. 2) The loss rebalancing prevents overfitting to the artificial missingness distribution. We will add this explanation of how our approach is tailored for missing data problems.