Principal Components Analysis based frameworks for efficient missing data imputation algorithms
Anonymous Author(s)
AffiliationAddress email
Abstract
Missing data is a commonly occurring problem in practice.
Many imputation methods have been developed to fill in the missing entries.
However, not all of themcan scale to high-dimensional data, especially the multiple imputation techniques.
Meanwhile, the data nowadays tending toward high-dimensional.
Therefore, in thiswork, we propose Principal Component Analysis Imputation (PCAI), a simple butversatile framework based on Principal Component Analysis (PCA) to speed upthe imputation process and alleviate memory issue of many available imputationtechniques, without sacrificing the imputation quality in term of MSE.
In addition,the frameworks can be used even when some or all of the missing features arecategorical, or when the number of missing features is large.
Next, we introduce
PCA Imputation - Classification (PIC), an application of PCAI for classificationproblem with some adjustment.
We validate our approach by experiments on various scenarios, which shows that PCAI and PIC can work with various imputationalgorithms, including the state-of-the-art ones and improve the imputation speedsignificantly, while achieving competitive mean square error/classification accuracycompared to direct imputation (i.e., impute directly on the missing data).
Introduction
Despite recent efforts in directly handling missing data [ missing data imputation approaches[5, 6, 7] remain commonly used.
This is because directly handling missing data can be complicatedand usually are developed for specific target problems or models, while imputation can be moreversatile.
Specifically, an important advantage of imputation is that the imputed data becomescomplete , i e., no longer have any missing values.
Therefore, it is easier to continue with otherpreprocessing steps, analysis and data visualizations.
Furthermore, one can deploy many modelsand choose the best one by using the available packages or software tools for non-missing data.
Meanwhile, directly handling missing data strategies do not have these advantages.
They are morecomplicated and not that readily available.
Many techniques have been developed for missing data imputation, ranging from traditional techniques such as MICE [5], K-Nearest Neighbors to recent machine learning/deep learning techniquessuch as GAIN [6], DL-GSA [7].
However, most of them are computationally expensive for bigdatasets.
For example, experiments in [8] show that under their experiment settings, for Fashion
MNIST [9], a dataset of 70,000 samples and 784 features, the MICE [5] and missForest [10] techniques are unable to finish the imputation process within three hours for a missing rate (the ratiobetween the number of missing entries versus the total number entries in the dataset) of 20%.
Sincedatasets nowadays are trending towards larger sizes [11], with hundreds of thousands of features[12], it is crucial to speed up the available imputation techniques.
Taking into account resource consumption and availability such speed up cannot be achieved by only providing more and betterhardware but by the development of new methods.
To achieve this goal, this work introduces two novel frameworks based on Principal Component
Analysis (PCA) to speed up the imputation process of many available techniques or the imputationclassification process for missing data classification problems.
The first framework, PCA Imputation(PCAI) is proposed to speed up the imputation speed by partitioning the data into the fully observedfeatures partition and the partition of features with missing data.
After that, the imputation of themissing part is performed based on the union of the PCA - reduced version of the fully observedpart and the missing part.
Interestingly, it turns out that the method has a great potential to aid theperformance of methods that rely on many parameters, such as Deep Learning imputation techniques.
Meanwhile, the second one, PCA Imputation - Classification (PIC) is proposed to deal with themissing data classification problems where dimension reduction is desirable in advance of the modeltraining step.
PIC is based on PCAI with some modifications.
Note that these frameworks aredifferent from the methods developed for principal component analysis under missing data presentedin [13, 14], which are about how to conduct PCA when the data contains missing values.
In summary, the contributions of this article are: (i) we introduce PCAI to improve the imputationspeed of many available imputation techniques; (ii) we introduce PIC to deal with missing dataclassification problems where dimension reduction is desirable; (iii) we analyze the potential strengthand drawbacks of these approaches; and (iv) we illustrate via experiments that our frameworkscan work with various imputation strategies while achieve comparable or even lower mean squareerror/higher classification accuracies compared to the corresponding original approaches, and alleviatethe memory issue in some approaches.
The rest of the paper is organized as follows.
In Section 2 and Section 3, we review some relatedwork in the field of missing data, and review two popular formulations of PCA.
Next, in Section4, Section 5, and Section 6, we introduce our novel PCAI and PIC frameworks, and study theirrelation to previous works, respectively.
After that, in Section 7, we demonstrate their capabilitiesvia experiments on various datasets.
The paper ends with conclusions, remarks, and future works in
Section 8.
Related Works
Various works have been published on missing data imputation to deal with different data analysissituations.
As an example, if one is interested in modeling the uncertainty associated with theimputation, suitable approaches can be multiple or Bayesian imputation techniques such as multipleimputations using Deep Denoising Autoencoders [15], Bayesian Principal Component Analysisbased imputation [16], and extreme learning machine multiple imputation [17].
In addition, graphicalmodels can be prominent candidates when transparency, estimability, and testability are desirable, andthese approaches can provide meaningful performance guarantees even if the missing values are not atrandom [18].
Next, for continuous data, matrix completion techniques such as Fast Alternating Least
Squares [19], softImpute [20] can quickly give good results.
In biology, the missing values are oftencategorical, and the imputed values need to be interpretable.
In such cases, classification techniquesor tree-based methods such as decision trees and fuzzy clustering with iterative learning (DIFC)
[21],missForest [10], the DMI algorithm [22], and sequential regression trees [23] are well-suited.
Inaddition, some recently developed methods that can handle mixed missing data are SICE [24], FEMI[25], and HCMM-LD [26].
When the sample sizes are large enough compared to the number offeatures, deep learning techniques such as Multiple Imputation Using Deep Denoising Autoencoders[15], DL-GSA [7], and Swarm Intelligence-Deep Neural Network [27] can be powerful imputers.
However, it is worth noting that deep learning methods usually require more data than statisticalimputation approaches.
Some other popularly used missing data imputation methods are multipleimputation by chained equation (MICE)
[5], K-nearest Neighbors imputation (KNNI)
[28], and meanimputation [28].
In addition, for the purpose of data imputation and data type, for classification, the impact ofimputation techniques on different classifiers may vary.
Specifically, [28] compares the performanceof logistic regression with regularization, k-nearest neighbours (kNN), random forest, classificationtree, and xgboost classifiers [28] on datasets with missing entries.
They use different imputationmethods (mean imputation/ MICE imputation [5]/ missForest [10]/ random imputation/ softImpute
[20]/ hot deck imputation, kNN imputation) and compare the performance.
According to the paper,mean imputation seems to outperform other counterparts for logistic regression with regularizationand kNN, random imputation wins for random forest, missForest seems to be the best imputer forclassification tree, and hot deck imputation is the best for xgboost.
With the rapid growth of data size [11, 12], it is necessary to speed up the available imputationmethods because many current approaches remain too slow for big datasets, as pointed out in anexample in Section 1.
This is where a popular dimension reduction method like PCA can come to use.
PCA projects the original higher-dimensional dataset into a representation of lower dimensionalityby extracting and retaining important information from the data and expressing this new informationbased on a set of orthogonal vectors known as principal components.
Its goal is to find lineartransformations of the original data that retain the maximal amount of variance.
Note that there aresome works on PCA under data missingness.
For example, [13] considers the problem of findingprincipal components as an optimization problem of an objective function and proposes iterativesolutions to it.
On the other hand, [29] proposes a multiple imputation method for the estimates of theparameters (components and axes) of PCA to take into account the variability due to missing values.
However, our work is different from these works in the sense that they target the problem of how toperform PCA for a dataset with missing data.
Meanwhile, our frameworks utilize PCA to speed upthe imputation processor to reduce the ratio between the number of features and the sample size.
Preliminaries
Let X =
[ x ij ] where i = 1 , ..., n ; j = 1 , ..., p be a input data matrix of n samples, p features.
Inaddition, assume that the features are centered and scaled.
We review two popular formulations of
PCA, which we refer to as PCA formulation 1 ( PCA-form1 ) and PCA formulation 2 ( PCA-form2 ).
3.1 PCA based on covariance matrix (PCA-form1)
Let Σ be the covariance matrix of X .
Next, let ( λ 1 , v 1 ) , ..., ( λ p , v p ) be the sorted eigenvalueeigenvector pairs of Σ such that λ 1 ≥ λ 2 ≥ ... ≥ λ p ≥ 0 .
Suppose that we choose the first r pairsfor dimension reduction.
Then the amount of variance explained by these r pairs is
[Equation]
In addition, let V =
[ v 1 , v 2 , ..., v r ] .
Then the dimension reduced version of X is XV .
3.2 PCA based on the input matrix X (
PCA
-form2)
The solution of PCA can also be produced based on the singular value decomposition of X [30]:
[Equation]
where U is an n × p orthogonal matrix, W is a p × p orthogonal matrix, and D is a p × p diagonalmatrix whose diagonal elements are d 1 ≥ d 2 ≥ ... ≥ d p ≥ 0 .
Suppose that r eigenvalues are used,then the projection matrix is V = W r W Tr where W r consists of the first r columns of W .
Thenthe dimension reduced version of X is also XV .
PCA Imputation (PCAI)
In this section, we detail our PCAI framework, a PCA based framework that is capable of significantlyimproving the imputation speed of an imputer for high dimensional data, alleviating the memoryissue for many approaches.
To start with some notations, let pca ( A ) be a function of a data matrix A .
The function returns( R A , V ) where R A is the PCA-reduced version of A , and V is the projection matrix where the i thcolumn of V is the eigenvector corresponding to the i th largest eigenvalue.
In addition, denote by
A ∪ B the columnwise concatenation of two data partition A and B of relevant sizes.
Next, supposethat we have a dataset D = F ∪ M , where F consists of data from fully observed features and Mconsists of data
The framework is as depicted in Algorithm 1.
We first conduct dimension reduction on the fullyobserved partition F , which produces a reduced version R of F .
Then, the imputation of M is doneon the set R ∪ M instead of D = F ∪ M as how imputations are usually done (i.e., impute directlyon the original missing data).
In conducting dimension reduction, we expect to reduce the dimensionof the fully observed partition so that the imputation of M can be faster.
Algorithm
Require:
F fully observed partition
M is the partition missing values
Procedure:
[Equation]
For the choice of the PCA formulation, note that if the number of samples is larger than the numberof features in F , then the size of the covariance matrix is smaller than the size of F .
Therefore, onemay expect using the formulation of PCA based on the covariance matrix, as in Section 3.1, to befaster.
Meanwhile, if the number of features in F is larger than the sample size, then the covariancematrix of F is larger than F .
Therefore, in such a case, it is better to use the PCA formulation basedon the data itself, i.e., formulation as in Section 3.2.
One may reckon that using R ∪ M instead of F ∪ M may lead to loss of information due todimension reduction and therefore lower the quality of imputation.
However, as will be illustratedin the experiments, the differences between the mean squared error of the imputed version versusthe ground truth for these approaches are only slightly different, and many times, PCAI seems to beslightly better.
This is possibly because PCA retains the important information from the data whileremoving some noise, and therefore helps improving the imputation quality.
However, PCAI also hassome shortcomings.
For problems where the sample size n is smaller than the number of features inthe fully observed block q , if PCA-form1 is used, the covariance matrix has the size of q × q , whichis bigger than the size n × q of the fully observed partition F .
This may make the PCA dimensionreduction process become computationally expensive, rendering PCAI to be slower than imputingdirectly on the original missing data.
This issue will be illustrated in the experiment section.
PCAI for classification (PIC)
In this section, we discuss a straightforward application of PCAI in classification, with a slightmodification for classification problems where it is desirable to conduct a dimension reduction beforetraining a model, such as when the number of features is much larger than the sample size.
Since PCAI conducts PCA on the fully observed partition F , it reduces the dimensions for a portionof the data.
Therefore, rather than imputing values using the PCAI framework and then conductinga dimension reduction step on F ∪ M ′ , one can perform dimension reduction on M ′ to get R ′ , a
PCA-reduced version of M ′ .
Then, one can use F ∪R ′ as reduced dimension data.
As will be shownin the experiments, this speeds up the imputation and classification process significantly.
This is thebasic idea of our Principle component Imputation for Classification (PIC) framework.
PIC operates as shown in Algorithm 2.
The procedure starts by performing PCA on the training fullyobserved partition F train , which gives the reduced version R train of F train and a projection matrix
V .
Next, we project F test on V to get the reduced version R test of F test .
Then, we impute M trainon R train ∪ M train to get the imputed version M ′ train .
Next, we impute M test on R test ∪ M testto get the imputed version M ′ test .
After that, if reduce miss is set to true, we perform dimensionreduction on M ′ train , M ′ test .
Then, we train the classifier on R train ∪ R ′ train , i.e., the union of thereduced version of F train and the reduced version of M train .
For prediction of a vector x ∈ D ,we can decompose x into x = ( x F , x M ) .
After that, we can project x F on V to get a projection r .
Similarly, we can project x M on V to a get projection r ′ .
Finally, we can predict the label of x usingthe classifier C with input ( r , r ′ ) .
Note that reduce miss is an option.
When the number of features in the missing partition M islarge, one may be interested in reducing the dimension of M ′ , and therefore, set reduce miss to True .
However, when the number of features in the missing partition is small, one may want to keep it to
False .
Also, since PIC is a straightforward application of PCAI for classification, the choice of PCAformulation should be used is similar to PCAI, which is analyzed in the previous section.
Algorithm framework
Require:
Procedure: ( R train , V ) ← pca
F
R test ← F test V
M ′ train ← imputed
M train
M ′ test ← imputed version of M on R test
M test if reduce miss then ( R ′ train , W ) ← pca ( M ′ train )
R ′ test ← M ′ test V
Train the classifier C on R
R
Classify based on R test ∪ R test , else
Train the classifier based on
M train
Classify based on R
M ′ test end if return trained classifier C
Relation to previous works
Various works have been done on PCA that are related to missing data, which mostly can becategorized into missing values imputation using PCA, or dimension reduction using PCA undermissing values.
Some typical works that make use of PCA for missing values imputation areprobabilistic PCA for missing flow volume data imputation [31]; chunk-wise iterative PCA fordata imputation on datasets with many samples[32]; [14] proposes a fast algorithm for PCA undermissing data that help in case of sparse, high dimensional data; [33] analyze maximum likelihood
PCA (MLPCA) on maximum likelihood missing data imputation; and [34] proposed an imputationapproach based on PCA and factorial analysis for mixed data.
Next, PCA under missing values was first studied in [35], where only one component and oneimputation iteration are used.
After that, [36] proposes a method based on MLPCA, where themethod assigns large variance to missing values prior to implementing the method, which aim toguide the algorithm to fit a PCA model disregarding those points.
Also, [37] introduce EM algorithmfor building a PCA model that can deal with missing data.
More recently, [38] proposes newtechniques for building a PCA model with missing data: known data regression (KDR), projection tothe model plane, KDR with principal component regression.
In addition, [39] studies estimation andimputation in Probabilistic PCA when the data is missing not at random.
Different from the previous approaches, PCAI is a framework to speed up the imputation process,which can be used with various imputation methods, including the aforementioned PCA imputationalgorithms and the state-of-the-art imputation algorithms such as softImpute [20], MissForest [10],
GAIN [6].
In addition, note that since PCAI and PIC conduct dimension reduction on the fullyobserved partition F , and not the missing portion M if reduce miss =
False , they can handle missingdata even if categorical features presents in the missing portion M , when being used with imputersthat’s capable of handling categorical/mixed data (MissForest [10], SICE [24], FEMI [25], etc.).
In
[Table]
addition, even if there exists categorical and continuous features in M ; or reduce miss =
True andthere exists categorical and continuous features in M , one can easily adjust the algorithm to conduct
PCA on continuous features only.
The previously mentioned PCA based approaches are, however,can only be used for continuous data, because PCA requires the data to be continuous.
Experiments
7.1 General experiment settings
We compare the speed (seconds) and MSE of PCAI with direct imputation (DI) , i.e., use animputation algorithm directly on the dataset.
The imputation approaches used for comparison:softImpute [20, 40], MissForest [10] 1 and Multiple Imputation by Chained Equation (MICE) [5, 41],kNN Imputation (KNNI), GAIN [6] are implemented with default configurations.
The codes will beavailable upon the acceptance of the paper.
For PIC, we compare the five fold cross-validation (CV)score (accuracy, speed) of PIC when dimension reduction is applied on the imputed missing part( PIC-reduce) , when dimension reduction is not applied on the imputed missing part ( PIC ), and when
PCA is applied to the imputed version on the full missing data ( DI-reduce ), and when no dimensionreduction is applied to imputed data after direct imputation ( DI ).
Here, the default PCA formulationis PCA-form1, unless specified otherwise.
For all PCA computation, the number of eigenvectors ischosen so that the minimum amount of variance explained is 95%.
Details of the datasets used in the experiments are available in Table 1.
All experiments are run onan AMD Ryzen 7 3700X CPU with 8 Cores, 16 processing threads, 3.6GHz, and 16GB RAM.Weterminate an experiment if no result is produced after 6,500 seconds of running or if there arises amemory allocating issue, and we denote this as NA in the result tables [Table]
7.2 Performance of PCAI and PIC when the missing values in M are randomly simulated
[Table]
Note that any datasets can be rearranged so that the first q features are not missing and the remaining ones are missing.
Therefore, without loss of generality, we assume that the first q features of each
[Table]
dataset are not missing, and the remaining ones contain missing value(s).
Then, we simulated missingdata randomly on the missing partition M with missing rates 20%, 40%
[Table]
and 60%.
Here, a missingrate of 20% means that 20% of the entries in the missing partition M are missing.
The results forsuch experiments are reported in Tables 2, 3, 4.
Due to space limit, the results related to PIC on
Fashion MNIST are reported in the Appendix.
[Table]
From the tables, it is clear that the proposed frameworks reduce the imputation time significantlywhile maintaining competitive MSE/classification accuracy compared to DI, in most of the cases.
For example, at the missing rate 20% on the Parkinson dataset (Table 4), when using GAIN forimputation, the running time of PIC-reduce(91.086s) is much lower compared to DI-reduce (130.349),the running time of PIC (89.984s) is also much lower compared to DI (129.702).
Another examplecan be seen from Table 2, for the Parkinson dataset, at 20% missing rate, when PCAI is applied tomissForest, the running time reduces to 188.324s, which is almost 1 / 5 of the DI (905.002s).
Next, on Fashion MNIST (Table 3), it is worth noticing that for MICE, DI cannot gives the results due tomemory issue but PCAI can alleviate this issue and deliver the results.
For KNNI, the running time for KNNI between the PCAI approach and direct imputation for
Parkinson (Table 2) is not much different.
However, for the Fashion MNIST dataset, KNNI using the
PCAI framework obviously deliver a competitive result in a significantly shorter time.
Specifically,
KNNI at a missing rate of 20% on Fashion MNIST gives a result after only 1607.850 seconds, while
DI takes up to 3,042.752 seconds.
This is because Fashion MNIST (70000 samples) has much moresamples than Parkinson (756 samples), and KNN need to do a lot of pairwise comparison.
Therefore,
PCAI and PIC would be extremely helpful for KNNI when the sample size and the number of fullyobserved features is large.
Note that it does not require the number of features with missing data tobe large or small.
From Table 2, we can see that PCAI generates a lot of improvements in MSE for GAIN, in additionto improvements in speed.
This is possibly because PCA reduces the number of features while thesample size remains the same, making such a deep learning approach more applicable to the newlyreduced data.
7.3 Performance on nonrandomly missing data
In many fields, the data are missing in a monotone pattern rather than random [44].
Therefore, wegenerate one-step monotone missing data on Fashion MNIST by first, randomly choose 20%, 40%,60% of the samples.
Then, we make them become missing by deleting the lower right corner bydeleting the intersection between the last 8 rows and the last 13 columns of each image array.
Theresults are reported in Table 5.
From the table, we can see that PIC-reduce is a great improvement inspeed compared to DI-reduce, and PIC is a significant improvement in speed compared to DI.
Thisillustrates that PIC can work effectively even for non-randomly missing data.
[Table]
7.4 PIC under different PCA formulations and number of missing features
The missing data in these experiments are generated at random as in Section 7.2 and the five foldcross validation results of SVM on the Gene dataset with q = 15000 , 20000 , are shown in Table 6and Table 7.
From these tables, one can see clearly that for datasets where the number of featuresare significantly higher than the number of samples such as Gene, PCA-form2, which is based onthe input data ( F specifically) gives much faster computations compared to PCA-form1, and also isfaster than direct imputation-classification without PCA.
In addition, when PCA-form1 is used, eventhough PIC and PIC-reduce are faster than PCA on directly imputed data (DI-reduce), they are stillmuch slower than direct imputation - classification without PCA.
Interestingly, the accuracy PIC and PIC-reduce are almost identical to PCA on directly imputed data,and are higher than direct imputation - classification without PCA.
Next, note that the main idea ofthe proposed methods is to reduce the dimension of the F to speed up the imputation.
Therefore, wehave made no assumption about the number of features in the missing portion M .
In Table 6 and
Table 7, q = 15000 , 20000 , which means 5,531 and 531 missing features in M , respectively.
Thisimplies PIC can handle datasets where M has many features.
[Table] [Table]
Conclusion and Remarks
We have presented two novel frameworks for datasets where many continuous features are fullyobserved, PCAI and PIC, that can speed up imputation algorithms significantly while having competitive accuracy MSE/accuracy compared to direct imputation and alleviate the memory issue forsome imputation approaches such as MICE, kNN.
In addition, the frameworks can be used evenwhen some or all of the missing features are categorical or when the number of missing featuresis large.
Note that when the sample size is significantly larger than the number of fully observedfeatures, PCA-form1 should be used since, in such a case, the covariance matrix is much smaller than
F , making it faster than PCA-form2.
On the other hand, when the number of fully observed featuresis significantly larger than the sample size, PCA-form2 should be preferred, as the covariance matrixis bigger than F itself in such a case.
A limitation of the proposed framework is that if there are notmany fully observed continuous features, then due to the computational cost of PCA, the proposedframeworks may not lead to any improvement in speed.
Even though PIC is only introduced for classification, the same strategy can be applied to a regressionproblem.
We would like to explore that in the future.
Moreover, since various dimension reductiontechniques such as sparse PCA [45], incremental PCA [46], truncated SVD [47] have been developedto suit different scenarios, it is worth investigating different dimension reduction techniques for PCAIand PIC.
In addition, it would be interesting to explore if applying a PCA variant to the missingpartition M would result in even a more efficient method for datasets with continuous features in themissing partition.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
See Table 2, Table 3, Table ??
Table
and Section7.
b) Have you read the ethics review guidelines and ensured that your
paper
conforms to
them?
[Yes] .
c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Did you describe the limitations of your work?
[Yes] .
See the last paragraph of Section4.
If
you are
including
theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] .
The code will
be available in the form of a Github repository upon acceptance, to preserve anonymity.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] .
See Section 7.
(c) Did you report error bars (e
g., with respect to the random
seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] .
See Section 7.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] .
See References list.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
The code can be found on a github repository upon the acceptance of the paper.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]