H AVE M ISSING D ATA ? M AKE I T M ISS M ORE ! I MPUTING T ABULAR D ATA WITH M ASKED A UTOENCODING
Anonymous authors
Paper under double-blind review
A BSTRACT
We present R E M ASKER , a novel method for imputing missing values in tabular data by extending the masked autoencoding framework.
In contrast to prior work, R E M ASKER is both simple – besides the missing values ( i e. , naturally masked), we randomly “re-mask” another set of values, optimize the autoencoder by reconstructing this re-masked set and apply the trained model to predict the missing values; and effective – with extensive evaluation on benchmark datasets, we show that R E M ASKER performs on par with or outperforms state-of-the-art methods in terms of both imputation fidelity and utility under various missingness settings, while its performance advantage often increases with the ratio of missing data.
We further explore theoretical justification for its effectiveness, showing that R E M ASKER tends to learn missingness-invariant representations of tabular data.
Our findings indicate that masked modeling represents a promising direction for further research on tabular data imputation.
The code is available at: https://anonymous.4open.science/r/remasker-5E3B
I NTRODUCTION
Missing values are ubiquitous in real-world tabular data due to various reasons during data collection, processing, storage, or transmission.
It is often desirable to know the most likely values of missing data before performing downstream tasks ( e.g., classification or synthesis).
To this end, intensive research has been dedicated to developing imputation methods (“imputers”) that estimate missing values based on observed data (Yoon et al., 2019; Jarrett et al., 2022; Kyono et al., 2021; Stekhoven & Buhlmann, 2012; Mattei & Frellsen, 2018).
Yet, imputing missing values in tabular data with high fidelity and utility remains an open problem, due to challenges including the intricate correlation across different features, the variety of missingness scenarios, and the scarce amount of available data with respect to the number of missing values.
The state-of-the-art imputers can be categorized as either discriminative or generative .
The discriminative imputers, such as MissForest (Stekhoven & Buhlmann, 2012), MICE (van Buuren &
Groothuis-Oudshoorn, 2011), and MIRACLE (Kyono et al., 2021), impute missing values by modeling their conditional distributions on the basis of other values.
In practice, these methods are often hindered by the requirement of specifying the proper functional forms of conditional distributions and adding the set of appropriate regularizers.
The generative imputers, such as GAIN (Yoon et al., 2019),
MIWAE (Mattei & Frellsen, 2018), GAMIN (Yoon & Sull, 2020), and HI-VAE (Nazabal et al., 2020), estimate the joint distributions of all the features by leveraging the capacity of deep generative models and impute missing values by querying the trained models.
Empirically, GAN-based methods often require a large amount of training data and suffer the difficulties of adversarial training (Goodfellow et al., 2014), while VAE-based methods often face the limitations of training through variational bounds (Zhao et al., 2022).
Further, some of these methods either require complete data during training or operate on the assumptions of specific missingness patterns.
In this paper, we present R E M ASKER , a novel method that extends the masked autoencoding (MAE) framework (Devlin et al., 2018;
He et al., 2022) to imputing missing values of tabular data.
The idea of R E M ASKER is simple: Besides the missing values in the given dataset ( i.e. , naturally masked), we randomly select and “re-mask” another set of values, optimize the autoencoder with the objective of reconstructing this re-masked set, and then apply the trained autoencoder to predict the missing values.
Compared with the prior work, R E M ASKER enjoys the following desiderata: ( i ) it is instantiated with Transformer (Vaswani et al., 2017) as its backbone, of which the self-attention mechanism is able to capture the intricate inter-feature correlation (Huang et al., 2020); ( ii ) without specific assumptions about the underlying missingness mechanisms, it is applicable to various scenarios even if complete data is unavailable; and ( iii ) as the re-masking approach naturally accounts for missing values and encourages learning high-level representations beyond low-level statistics, R E M ASKER works effectively even under a high ratio of missing data ( e.g. , 0.7).
With extensive evaluation on 12 benchmark datasets under various missingness scenarios, we show that R E M ASKER performs on par with or outperforms 13 popular methods in terms of both imputation fidelity and utility, while its performance advantage often increases with the ratio of missing data.
We further explore the theoretical explanation for its effectiveness.
We find that R E M ASKER encourages learning missingness-invariant representations of tabular data, which are insensitive to missing values.
Our findings indicate that, besides its success in the language and vision domains, masked modeling also represents a promising direction for future research on tabular data imputation.
R ELATED W ORK
Here, we survey relevant literature in three categories.
Tabular data imputation.
The existing imputation methods can be roughly categorized as either discriminative or generative.
The discriminative methods (Stekhoven & Buhlmann, 2012; van
Buuren & Groothuis-Oudshoorn, 2011; Kyono et al., 2021) often specify a univariable model for each feature conditional on all others and perform cyclic regression over each target variable until convergence.
Recent work has also explored adaptively selecting and configuring multiple discriminative imputers (Jarrett et al., 2022).
The generative methods either implicitly train imputers as generators within the GAN framework (Yoon et al., 2019; Yoon & Sull, 2020) or explicitly train deep latent-variable models to approximate the joint distributions of all features (Mattei & Frellsen, 2018; Nazabal et al., 2020).
There are also imputers based on representative-value ( e.g. , mean, median, or frequent values) substitution (Hawthorne & Elliott, 2005), EM optimization (García-Laencina et al., 2010), matrix completion (Hastie et al., 2015), or optimal transport (Muzellec et al., 2020).
Transformer.
Transformer has emerged as a dominating design (Vaswani et al., 2017) in the language domain, in which multi-head self-attention and MLP layers are stacked to capture both short- and long-term correlations between words.
Recent work has explored the use of Transformer in the vision domain by treating each image as a grid of visual words (Dosovitskiy et al., 2020).
For instance, it has been integrated into image generation models (Jiang et al., 2021; Zhang et al., 2021; Hudson &
Zitnick, 2021), achieving performance comparable to CNN-based models.
Masked autoencoding.
Autoencoding is a classical method for learning representation in a selfsupervised manner (Vincent et al., 2008; Pathak et al., 2016): an encoder maps an input to its representation and a decoder reconstructs the original input.
Meanwhile, masked modeling is originally proposed as a pre-training method in the language domain: by holding out a proportion of a word sequence, it trains the model to predict the masked words (Devlin et al., 2018;
Radford & Narasimhan, 2018).
Recent work has combined autoencoding and masked modeling in vision tasks (Dosovitskiy et al., 2020; Bao et al., 2022).
Particularly, the seminal MAE (He et al., 2022) represents the state of the art in self-supervised pre-training on the ImageNet-1K benchmark.
The work is also related to that models missing data by adapting existing model architectures (Przewi˛e´zlikowski et al., 2021).
To our best knowledge, this represents the first work to explore the masked autoencoding method with Transformer in the task of tabular data imputation.
R E M ASKER
Next, we present R E M ASKER , an extremely simple yet effective method for imputing missing values of tabular data.
We begin by formalizing the imputation problem.
3. P ROBLEM F ORMALIZATION
Incomplete data.
To model tabular data with d features, we consider a d -dimensional random variable x ≜ ( x 1 , . . .
, x d ) ∈
X 1 × . . .
× X d , where X i is either continuous or categorical for i ∈ { 1 , . . .
, d } .
[Table]
Embedding Padded Embedding
[Table]
+F8+1+usIy1sY4uNHWl8knlpqprQwHLFca1iKuO5e3EhLQKpJggBVoZfxjARVgAFj7vc4HcotRam8BxAWmg9n6I1yeAdzvgNBJvQej7Jy5nvcxcmVOSoUcjncL9t/9JtN1wj3fT+tjg/GqTvB28/H/VOP6zuss2esefsFUvZMTtln9gZGzJglv1gP9mviEUvozfR4RKNOquefbYW0ckfTrnEdQ==</latexit> [ ⇤ ] The observational access to x is mediated by an mask variable m ≜ ( m 1 , . . .
, m d ) ∈ { 0 , 1 } d , which indicates the missing values of x , such that x i is accessible only if m i
= 1 .
In other words, we observe x in its incomplete form ˜ x ≜ (˜ x 1 , . . . ,
˜ x d ) with
[Equation]
where ∗ denotes the unobserved value.
Missingness mechanisms.
Missing values occur due to various reasons.
To simulate different scenarios, following the prior work (Yoon et al., 2019; Jarrett et al., 2022), we consider three missingness mechanisms: MCAR (“missing completely at random”)
– the missingness does not depend on the data, which indicates that ∀ m , x , x ′ , p ( m | x ) =
p ( m | x ′ ) ; MAR (“missing at random”) – the missingness depends on the observed values, which indicates that ∀ m , x , x ′ , if the observed values of x and x ′ are the same, then p ( m | x ) = p ( m | x ′ ) ; and MNAR (“missing not at random”) – the missingness depends on the missing values as well, which is the case if the definitions of MCAR and MAR do not hold.
In general, it is impossible to identify the missingness distribution of MNAR without domain-specific assumptions or constraints (Ma & Zhang, 2021).
Imputation task.
In this task, we are given an incomplete dataset D ≜ { (˜ x ( i ) , m ( i ) ) } ni =1 , 1 which consists of n i.i.d.
realizations of ˜ x and m .
The goal is to recover the missing values of each input ˜ x by generating an imputed version ˆ x ≜ (ˆ x 1 , . . .
, ˆ x d ) such that
[Equation]
where ¯ x i is the imputed value.
3. D ESIGN OF R E M ASKER
The R E M ASKER imputer extends the masked autoencoding (MAE) framework (Dosovitskiy et al., 2020;
Bao et al., 2022; He et al., 2022) that reconstructs masked components based on observed components.
As illustrated in Figure 1, R E M ASKER comprises an encoder that maps the observed values to their representations and a decoder that reconstructs the masked values from the latent representations.
However, unlike conventional MAE, as the data in the imputation task is inherently incomplete ( i.e. , naturally masked), we employ a “re-masking” approach that explicitly accounts for this incompleteness in applying masking and reconstruction.
At a high level, R E M ASKER works in two phases: fitting – it optimizes the model with respect to the given dataset, and imputation – it applies the trained model to predict the missing values of the dataset. Re-masking.
In the fitting phase, for each input ˜ x , in addition to its missing values, we also randomly select and mask out another subset ( e.g. , 25%) of ˜ x ’s values.
Formally, letting m be ˜ x ’s mask,
we // fitting phase
D do ; // remasking z ← f θ x m m ) ; // encoding unmasked values pad z tokens; update
; // minimizing reconstruction loss // imputation phase 7 foreach
D do z ←
; // encoding observed values pad
; ¯ m ←
; // predicting missing values ˆ x ← ˜ x m
; define another mask vector m ′ ∈ { 0 , 1 } d , which is randomly sampled without replacement, following a uniform distribution.
Apparently, m and m ′ entail three subsets: I
[Equation]
remask
[Equation]
[Equation]
[Equation]
= 0 } I unmask =
{ i | m
[Equation]
=
[Equation]
m ′ [Equation] = 1 } Let ˜ x m , ˜ x m ∧ m ′ , and ˜ x m ∧ m ′ respectively be the masked, re-masked, and unmasked values.
With a sufficient number of re-masked values, in addition to the missing values, we create a challenging task that encourages the model to learn missingness-invariant representations (more details in § 5).
Note that in the imputation phase, we do not apply re-masking.
Encoder.
The encoder embeds each value using an encoding function and processes the resulting embeddings through a sequence of Transformer blocks.
In implementation, we apply linear encoding function to each value x : enc ( x ) =
w x + b , where w and b are learnable parameters.
2 We also add positional encoding to x ’s embedding to force the model to memorize x ’s position in the input ( e.g. , the k -th feature): pe ( k, 2 i )
= sin( k/ 10000
i/d ) , where k and i respectively denote x ’s position in the input and the dimension of the embedding, and d is the embedding width.
Note that the encoder is only applied to the observed values: in the fitting phase, it operates on the observed values after re-masking ( i.e. , the unmasked set I unmask ); in the imputation phase, it operates on the non-missing values ( i.e. , the union of re-masked and unmasked sets I unmask ∪ I remask ), as illustrated in Figure 1.
Decoder.
The R E M ASKER decoder is instantiated as a sequence of Transformer blocks followed by an MLP layer.
Different from the encoder, the decoder operates on the embeddings of both observed and masked values.
Following (Devlin et al., 2018; He et al., 2022), we use a shared, learnable mask token as the initial embedding of each masked value.
The decoder first adds positional encoding to the embeddings of all the values (observed and masked), processes the embeddings through a sequence of Transformer blocks, and finally applies linear projection to map the embeddings to scalar values as the predictions.
Similar to (He et al., 2022), we use an asymmetric design with a deep encoder and a shallow decoder ( e.g. , 8 blocks versus 4 blocks), which often suffices to re-construct the masked values.
Conventional MAE focuses on representation learning and uses the decoder only in the training phase.
In R E M ASKER , the decoder is required to re-construct the missing values and is thus used in both fitting and imputation phases.
Reconstruction loss.
Recall that the R E M ASKER decoder predicts the value for each input feature.
We define the reconstruction loss functions as the mean square error (MSE) between the reconstructed and original values on ( i ) the re-masked set
I remask and ( ii ) unmasked set I unmask .
We empirically experiment with different reconstruction loss functions ( e.g. , only the re-masked set or both re-masked and unmasked sets).
Putting everything together, Algorithm 1 sketches the implementation of R E M ASKER .
E VALUATION
We evaluate the empirical performance of R E M ASKER in various scenarios using benchmark datasets.
Our experiments are designed to answer the following key questions: ( i ) Does R E M ASKER work? –
We compare R E M ASKER with a variety of state-of-the-art imputers in terms of imputation quality.
( ii ) How does it work?
– We conduct an ablation study to assess the contribution of each component of R E M ASKER to its performance.
( iii )
What is the best way of using R E M ASKER ?
– We explore the use of R E M ASKER as a standalone imputer as well as one component of an ensemble imputer to understand its best practice.
Datasets.
For reproducibility and comparability, similar to the prior work (Yoon et al., 2019; Jarrett et al., 2022), we use 12 real-world datasets from the UCI Machine Learning repository (Dua & Graff, 2017) with their characteristics deferred to Appendix § A.1.
Missing mechanisms.
We consider three missingness mechanisms.
In MCAR, the mask vector of each input is realized following a Bernoulli random variable with a fixed mean.
In MAR, with a random subset of features fixed to be observable, the remaining features are masked using a logistic model.
In MNAR, the input features of MAR are further masked following a Bernoulli random variable with a fixed mean.
We use the HyperImpute platform (Jarrett et al., 2022) to simulate the above missing mechanisms.
Baselines.
We compare R E M ASKER with 13 state-of-the-art imputation methods: HyperImpute (Jarrett et al., 2022), a hybrid imputer that performs iterative imputation with automatic model selection;
MIWAE (Mattei & Frellsen, 2018), an autoencoder model that fits missing data by optimizing a variational bound; EM (García-Laencina et al., 2010), an iterative imputer based on expectationmaximization optimization; GAIN (Yoon et al., 2019), a generative adversarial imputation network that trains the discriminator to classify the generator’s output in an element-wise manner; ICE, an iterative imputer based on regularized linear regression; MICE, an ICE-like, iterative imputer based on Bayesian ridge regression; MIRACLE (Kyono et al., 2021), an iterative imputer that refines the imputation of a baseline by simultaneously modeling the missingness generating mechanism; MissForest (Stekhoven & Buhlmann, 2012), an iterative imputer based on random forests; Mean (Hawthorne & Elliott, 2005), Median, and Frequent, which impute missing values using column-wise unconditional mean, median, and the most frequent values, respectively; Sinkhorn (Muzellec et al., 2020), an imputer trained through the optimal transport metrics of Sinkhorn divergences; and SoftImpute (Hastie et al., 2015), which performs imputation through soft-thresholded singular value decomposition.
Metrics.
For each imputation method, we evaluate the imputation fidelity and utility by comparing its imputed data with the ground-truth data.
In terms of fidelity, we mainly use two metrics: root mean square error (RMSE) to measure how the individual imputed values match the ground-truth data, and the Wasserstein distance (WD) to measure how the imputed distribution matches the ground-truth distribution.
In terms of utility, we use area under the receiver operating characteristic curve (AUROC) as the metric on applicable datasets ( i.e. , ones associated with classification tasks).
In the case of multi-class classification, we use the one versus rest (OvR) setting.
To be fair, we use logistic regression as the predictive model across all the cases.
4. O VERALL P ERFORMANCE
We evaluate R E M ASKER and baseline imputers on the benchmark datasets under the MAR setting with 0.3 missingness ratio, with results summarized in Figure 2.
Observe that R E M ASKER consistently outperforms all the baselines in terms of both fidelity (measured by RMSE and WD) and utility (measured by AUROC) across all the datasets.
Recall that the benchmark datasets are collected from a variety of domains with highly varying characteristics ( cf.
Table 5): the dataset size varies fromto 20,060, while the number of features ranges from 7 to 57.
Its superior performance across all the datasets demonstrates that R E M ASKER effectively models the intricate correlation among different features, even if the amount of available data is scarce.
The only imputer with performance close to R E M ASKER is HyperImpute (Jarrett et al., 2022), which is an ensemble method that integrates multiple imputation models and automatically selects the most fitting model for each column of the given dataset.
This highlights that the modeling capacity of R E M ASKER ’s masked autoencoder is comparable with ensemble models.
In Appendix § B.1, we conduct a more comprehensive evaluation
[Figure]
O
C
[Figure]
M SE
[Figure] [Figure]
by simulating all three missingness scenarios (MCAR, MAR, and MNAR) with different missingness ratios.
The results show that R E M ASKER consistently performs better across a range of settings.
4. S ENSITIVITY A NALYSIS
To assess the factors influencing R E M ASKER ’s performance, we conduct sensitivity analysis by varying the dataset size, the number of features in the dataset, and the missingness ratio under the
MAR setting.
Figure 3 shows the performance of R E M ASKER within these experiments against the six closest competitors (HyperImpute, ICE, MissForest, GAIN, MIWAE, and Sinkhorn) on the letter dataset.
We have the following observations.
( a ) The performance of R E M ASKER improves with the size of available data, while its advantage over other imputers (with the exception of HyperImpute) grows with the dataset size.
( b ) The number of features has a significant impact on the performance of R E M ASKER , with its advantage over other imputers increasing steadily with the number of features.
This may be explained by that R E M ASKER relies on learning the holistic representations of inputs, while including more features contributes to better representation learning.
( c ) R E M ASKER is fairly insensitive to the missingness ratio.
For instance, even with 0.7 missingness ratio, it achieves RMSE below 0.1, suggesting that it effectively fits sparse datasets.
In Appendix § B.2, we also conduct an evaluation on other datasets with similar observations.
4. A BLATION S TUDY
We further conduct an ablation study of R E M ASKER to understand the contribution of different components to its performance using the letter dataset.
The results on other datasets are deferred to Appendix § B.3.
[Table]
Model design.
The encoder and decoder of R E M ASKER can be flexibly designed.
Here, we study the impact of three key parameters, the encoder depth (the number of Transformer blocks in the encoder), the embedding width (the dimensionality of latent representations), and the decoder depth, with results summarized in Table 1a, Table 1b, and Table 1c, respectively.
Observe that the performance of R E M ASKER reaches its peak with a proper model configuration (encoder depth = 8, decoder depth = 8, and embedding width = 64).
This observation suggests that the model complexity needs to fit the given dataset: it needs to be sufficiently complex to effectively learn the holistic representations of inputs but not overly complex to overfit the dataset.
We also compare the performance of R E M ASKER with different backbone models ( i.e. , Transformer, linear, and convolutional) with the number of layers and the size of each layer fixed as the default setting. As shown in Table 2, Transformer-based R E M ASKER largely outperforms the other variants, which may be explained by that the self-attention mechanism of Transformer is able to effectively capture the intricate inter-feature correlation under limited data Huang et al.
[Table] [Table]
Reconstruction loss.
We define the reconstruction loss as the error between the reconstructed and original values on the re-masked values I mask+ and the unmasked values I unmask .
We measure the performance of R E M ASKER under three different settings of the construction loss: ( i ) I mask+ ∪ I unmask , ( ii ) I mask+ only, and ( iii ) I unmask only on the letter and california datasets, with results shown in Table 3.
Observe that using the reconstruction of unmasked values only is insufficient and yet including the reconstruction loss of unmasked values improves the performance, which is especially the case on the california dataset.
This finding is different from the vision domain in which computing the loss on unmasked image patches reduces accuracy (He et al., 2022).
We hypothesize that this difference is explained as follows.
Unlike conventional MAE, due to the naturally missing values in tabular data, relying on re-masked values provides limited supervisory signals.
Moreover, while images are signals with heavy spatial redundancy ( i.e. , a missing patch can be recovered from its neighboring patches), tabular data tends to be highly semantic and informationdense.
Thus, including the construction loss of unmasked values improves the model training [Figure]
4. P RACTICE OF R E M ASKER
Finally [Figure] we explore the optimal practice of R E M ASKER .
[Figure]
Training regime.
The ablation study above by default uses 600 training epochs.
Figure 4(a) shows the impact of training epochs, in which we vary the training epochs from 100 to 1,600 and measure the performance of R E M ASKER on the letter dataset.
Observe that the imputation performance improves (as RMSE and WD decrease and AUROC increases) steadily with longer training and does not fully saturate even at 1,600 epochs.
However, for efficient training, it is often acceptable to terminate earlier ( e.g. , 600 epochs) with sufficient imputation performance.
To further validate the trainability of R E M ASKER , with the maximum number of training epochs fixed at 600 (which affects the learning rate scheduler), we measure the reconstruction loss as a function of the training epochs.
As shown in Figure 4(b), the loss quickly converges to a plateau within about 100 epochs and steadily decreases after that, demonstrating the trainability of R E M ASKER .
Masking ratio.
The masking ratio controls the number of re-masked values (after excluding missing values).
Table 4a shows its impact on the performance of R E M ASKER .
Observe that the optimal ratio differs across different datasets, which may be explained by the varying number of features of different datasets (16 versus 9 in letter and california ).
Intuitively, a larger number of features affords a higher masking ratio to balance ( i ) encouraging the model to learn missingnessinvariant representations and ( ii ) having sufficient supervisory signals to facilitate the training.
[Table]
Standalone vs. ensemble.
Besides using R E M ASKER as a standalone imputer, we explore its use as a base imputer within the ensemble imputation framework of HyperImpute, with results summarized in Table 4b.
It is observed that compared with the default setting (with mean substitution as the base imputer), using R E M ASKER as the base imputer improves the imputation performance, suggesting another effective way of operating R E M ASKER .
D ISCUSSION
The empirical evaluation above shows R E M ASKER ’s superior performance in imputing missing values of tabular data.
Next, we provide theoretical justification for its effectiveness.
By extending the siamese form of MAE (Kong & Zhang, 2022), we show that R E M ASKER encourages learning missingness-invariant representations of input data, which requires a holistic understanding of the data even in the presence of missing values.
Let f θ ( · ) and d ϑ ( · ) respectively be the encoder and decoder.
For given input x , mask m , and re-mask m ′ , the reconstruction loss of R E M ASKER training is given by (here we focus on the reconstruction of re-masked values):
[Equation]
where ⊙ denotes element-wise multiplication.
Let m + ≜ m ⊙ m ′ and m − ≜ m ⊙ (1 − m ′ ) .
Eq (3) can be simplified as: ℓ ( x , m + , m − ) =
∥
d ϑ ( f θ ( x ⊙ m + )) ⊙ m − − x ⊙ m − ∥ 2 .
As the embedding dimensionality is typically much larger than the number of features, it is possible to make the autoencoder lossless.
In other words, for a given encoder f θ ( · ) , there exists a decoder d ϑ ′ ( · ) , such that d ϑ ′ ( f θ ( x ⊙ m − )) ⊙ m − ≈ x ⊙ m − .
We can further re-write Eq (3) as:
[Equation]
We now define a new distance metric ∆ ϑ,ϑ
[Equation]
z ) − d ϑ ′ ( z ′ )) ⊙ m − ∥ 2 .
Then, Eq (3) is reformulated as:
[Equation]
Note that optimizing Eq (5) essentially minimizes the difference between x ’s representations under m + and m − (with respect to the decoder).
As m + and m − mask out different values, this formulation promotes learning representations insensitive to missing values.
[Figure]
To validate the analysis above, we empirically measure the CKA similarity (Kornblith et al., 2019) between the latent representations ( i.e. , the output of R E M ASKER ’s encoder) of complete inputs and inputs with missing values, with results shown in Figure 5.
Observe that the CKA measures under different missingness ratios all steadily increase with the training length, indicating that
R E M ASKER tends to learn missingness-invariant representations of tabular data, which may explain for its imputation effectiveness.
C ONCLUSION
In this paper, we conduct a pilot study exploring the masked autoencoding approach for tabular data imputation.
We present R E M ASKER , a novel imputation method that learns missingness-invariant representations of tabular data and effectively imputes missing values under various scenarios.
With extensive evaluation on benchmark datasets, we show that R E M ASKER outperforms state-of-the-art methods in terms of both imputation utility and fidelity.
Our findings indicate that masked tabular modeling represents a promising direction for future research on tabular data imputation.
A E XPERIMENTAL D ETAILS A. D ATASETS
Table 5 summarizes the characteristics of the datasets used in our experiments.
A. P ARAMETER S ETTING
The default parameter setting of R E M ASKER is listed in Table 6.
[Table]
[Table]
B A DDITIONAL R ESULTS B. O VERALL P ERFORMANCE
Figure 6, 7, and 8 respectively show the imputation performance of R E M ASKER and 8 baselines on 12 benchmark datasets under the MAR, MCAR
[Table]
and MNAR scenarios with the missingness ratio varying from 0.1 to 0.7.
Observed that R E M ASKER performs on par with or outperforms almost all the baselines across a wide range of settings.
Note that the MIRACLE imputer does not work on the Compression dataset and the Raisin dataset under some settings, of which the results are not reported.
Given that both Compression and Raisin are relatively small datasets, one possible explanation is that MIRACLE requires a sufficient amount of data to train the model. Why does R E M ASKER generalize across the settings of MAR, MCAR [Table]
and MNAR? One possible explanation is as follows. Recall that in MCAR, the mask vector of each input is realized following a
Bernoulli random variable with a fixed mean; in MAR, with a random subset of features fixed to be observable, the remaining features are masked using a logistic model; in MNAR, the input features of MAR are further masked following a Bernoulli random variable with a fixed mean. Regardless of the missingness mechanism, it is rare that the values of one feature x are missing across all the records. Thus, by its design, R E M ASKER is able to learn to re-construct feature x i conditional on other features x ¯ i =
( x 1 , . .
, x i − 1 , x i +1 , . . .
, x d ) .
Yet, as reflected in the imputation results, the learning to re-construct performs better under MCAR, in which the missing values are evenly distributed across different features, than MAR or MNAR, in which the missing values are not evenly distributed.
[Figure]
Figure 9 shows the sensitivity analysis of R E M ASKER and other 6 baselines on the california dataset under the MAR, MCAR, and MNAR settings.
The observed trends are generally similar to that in Figure 3, which further demonstrates the observations we made in § 4 about how different factors may impact R E M ASKER ’s imputation performance.
B. A BLATION S TUDY
The ablation study of R E M ASKER on the california dataset is shown in Table 7.
Observed that the performance of R E M ASKER reaches its peak with encoder depth = 6, decoder depth = 4, and embedding width = 32.
[Figure]
B. T RAINING R EGIME
Figure 10 shows the imputation performance of R E M ASKER on the california dataset when the training length varies from 100 to 1,600 epochs.
Figure 11 plots the convergence of reconstruction loss in R E M ASKER , showing a trend similar to Figure 4(b).
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Figure] [Figure]
[Figure] [Figure]