Probabilistic Missing Value Imputation for Mixed Categorical and Ordered Data
Anonymous Author(s)
AffiliationAddress email
Abstract
Many real-world datasets contain missing entries and mixed data types includingcategorical and ordered (e.g. continuous and ordinal) variables.
Imputing themissing entries is necessary, since many data analysis pipelines require completedata, but challenging especially for mixed data.
This paper proposes a probabilisticimputation method using an extended Gaussian copula model that supports bothsingle and multiple imputation.
The method models mixed categorical and ordereddata using a latent Gaussian distribution.
The unordered characteristics of categorical variables is explicitly modeled using the argmax operator.
The method makesno assumptions on the data marginals nor does it require tuning any hyperparameters.
Experimental results on synthetic and real datasets show that imputationwith the extended Gaussian copula outperforms the current state-of-the-art for bothcategorical and ordered variables in mixed data.
Introduction
Modern datasets from healthcare, the sciences, and the social sciences often contain missing entriesand mixed data types such as continuous, ordinal, and categorical.
Social survey datasets, forexample, are typically mixed because they include variables like age (continuous), demographicgroup (categorical), and Likert scales (ordinal) measuring how strongly a respondent agrees withcertain stated opinions.
Continuous variables are encoded as real numbers and sometimes callednumeric.
we refer to variables that admit a total order (e g. continuous and ordinal) as orderedvariables.
In contrast, a categorical variable, also called nominal, can take one of a fixed number ofunordered values such as “A", “B", “AB", or "O" for blood type.
Most data analysis techniques require a complete dataset, so missing data imputation is an essentialpreprocessing step.
It is also often of interest to propagate imputation uncertainty into subsequentanalyses through multiple imputation, which generates several potentially different imputed datasets[24].
An imputation method should ideally use all the collected data—regardless of data type—to impute any missing data entries.
However, most imputation approaches, whether explicitly orimplicitly, assume that each variable admits a total order and, as a result, cannot impute categoricalvariables without proper preprocessing.
There is no satisfying, successful, and widely adopted method for imputing categorical variables,especially in mixed datasets.
It is tempting to reduce categorical imputation to ordinal imputationusing an integer encoding in which each category is assigned a number; however, this encodingrequires choosing an arbitrary ordering for the categories that may affect the results of downstream(e.g., predictive) models.
(This problem is most severe for linear and neural network models, whereastree-based models are less sensitive [29].)
Instead, it is more common to use one-hot encoding torepresent a categorical variable using a binary vector with one entry for each category.
With thisencoding, imputation methods developed for continuous or binary data, such as the popular low rank matrix completion methods (LRMC) [23, 16, 27], can be jerry-rigged for categorical imputation.
However, LRMC is only proper when the encoded matrix (both ordered and encoded categoricalvariables) is approximately low rank.
It has been observed that LRMC performs poorly on longskinny (many samples n and few features p ) datasets because the low rank assumption fails [8, 32].
Iterative imputation methods including MICE [4] and missForest [26] can directly operate on theunordered categories.
MICE learns the conditional distribution of each variable (whether orderedor unordered) using all other variables via linear and logistic regression.
However, the learnedconditional models may be incompatible in the sense that the joint distribution cannot exist [4].
MissForest uses random forest to predict each variable and yields more accurate single imputations[26], but cannot provide multiple imputation.
Both methods converge slowly (or sometimes diverge)for large datasets because they train many models on (possibly) ill-conditioned data.
This paper expands on the idea of modeling categorical data and multivariate data interaction using alatent continuous space [10, 5].
This approach can explicitly modeling the categorical distribution,which is critical for providing multiple imputation.
This paper models the latent space as a Gaussiandistribution and models each categorical variable as the argmax of a latent Gaussian vector.
Thischoice is inspired by the Gaussian copula model [13, 7, 9], which yields high quality imputationsfor ordered data [32].
However, it cannot be used when the data contains categorical variables.
Thispaper proposes the extended Gaussian copula to overcome this limitation by explicitly modeling eachdata type.
In contrast, other imputation methods typically require bespoke preprocessing for the inputdata to perform well.
For mixed categorical and ordered data, the extended Gaussian copula modelgenerates each categorical variable using a latent Gaussian vector and each ordered variable using alatent Gaussian scalar.
The latent Gaussian correlations capture the dependence structure.
Contributions.
This paper makes three main contributions to the literature:
A probabilistic model, the extended Gaussian copula , for mixed data including continuous, ordinal,and categorical variables.
The model is free of hyperparameters and makes no assumptions on themarginal distribution of each data type.2.
A single imputation method that empirically provides state-of-the-art accuracy for mixed data anda multiple imputation method that can quantify imputation uncertainty by measures such as thecategory probability for categorical variables and confidence intervals for continuous variables.3.
A robust and efficient parameter fitting algorithm for the extended Gaussian copula that supports
further acceleration through parallelization, mini-batch training, and low rank structure.
Related work.
Modeling a categorical variable as the argmax of a latent continuous vector is aclassical technique used in the multinomial probit model [3, 21] in the context of supervised learning.
Instead, our model is more similar to [5], which proposed one way to accommodate categoricalvariables in the Gaussian copula.
However, the model in [5] has many redundant parameters and thusis unidentifiable, while the extended Gaussian copula is identifiable and can match any categoricalmarginal distribution.
Methodology
Notation.
For f : R K → R , we define the (possibly set-valued) pre-image of f as f − 1 ( y ) :
= { x :f ( x ) =
y } .
We define argmax( z ) of a vector z =
( z 1 , . . .
, z p ) as the index of the maximum entryso that z argmax( z ) = max i =1 ,...,p z i .
We use N ( µ , Σ) to denote the multivariate normal distributionwith mean µ and covariance Σ .
We use 0 to denote the all-zero vector and 1 for the all-one vector,where the context determines the length of the vector.
We introduce our model in Section 2.1, then derive the model-based imputation methods in Section 2.2and finally present model estimation algorithms in Section 2.3.
Throughout the paper, we assumethe missing complete at random (MCAR) mechanism
2.1 Extended Gaussian copula with categorical variables
We first show how to model a categorical variable by transforming a latent Gaussian vector.
Then weextend this model to generate categorical vectors and mixed categorical and ordered vectors.
To easethe notation, we assume that all categorical variables have K categories encoded as { “1” , . . .
, “ K ” } .
It is straightforward to allow categorical variables with different numbers of categories.
2.1.1 Univariate categorical variable
We model a univariate categorical variable x with K categories as the argmax of a K -dim latent
Gaussian z = ( z 1 , . . .
, z K ) with some mean µ = ( µ 1 , . . .
, µ K ) and identity covariance.
That is,
[Equation]
We call the distribution in Eq.
(1) the Gaussian-Max distribution.
Without loss of generality, weassume µ 1 = 0 , as the argmax is invariant under translations, i.e., µ ← µ + α 1 for α ∈ R .
While adense covariance matrix is sometimes used for z [5], we prefer the model of Eq.
(1) as it is identifiable.
Theorem 1 states that any categorical distribution corresponds to a unique choice of µ under Eq.
All proofs are in the supplement.
In Section 2.3.1 we describe an algorithm to estimate µ for a givencategorical distribution.
Theorem 1 (Existence and Uniqueness) .
For any categorical distribution P [ x = “ k ”] = p k > 0 fork = 1 , . . .
, K such that (cid:80)
Kk =
1 p k = 1 , there is a unique µ ∈ R K with µ 1 = 0 such that
[Equation]
I K
[Equation] [Equation]
2.1.2 Multivariate categorical vector
We model a categorical vector x = ( x 1 , . . .
, x p ) by supposing each of its entries x j follows the
Gaussian-Max distribution.
That is, x j = argmax( z ( j ) + µ ( j ) )
for some µ ( j ) and isotropic Gaussianz ( j ) .
Additionally, the latent Gaussian variables z ( j ) corresponding to different categorical variablesmay be correlated.
We call this model the categorical latent Gaussian (CLG) model (see Definition 1).
For z :=
( z (1) , . . .
, z ( p ) ) , we use [ j ] to denote the indices of z ( j ) in z , so z
[ j ] =
z ( j ) for j = 1 , . .
. , p .
Definition 1 (Categorical latent Gaussian) .
For a categorical vector x = ( x 1 , . . .
, x p ) , we say xfollows the categorical latent Gaussian x ∼ CLG (Σ , µ ) , if there exists a correlation matrix Σ and µsuch that (1) for z ∼ N ( 0 , Σ) , x j = argmax( z [ j ] + µ [ j ] ) ; (2) Σ
[ j ] , [ j ] = I K , for every j = 1 , . . .
, p .
In the CLG model, the value of µ [ j ] suffices to determine the marginal distribution of each categoricalx j , as the marginal distribution of z [ j ] is N ( 0 , I K ) and independent of Σ .
The correlation matrix Σintroduces dependencies between different categorical variables in x .
Consider two categorical variables x j 1 and x j 2 whose joint distribution is described by P ∈ R K × K ,where p kl
= P [ x j 1 = k, x
j 2 = l ] .
The CLG model captures the dependence between x j 1 and x j 2 bythe correlation submatrix Σ [ j 1 ] , [ j 2 ] .
Note that P has only ( K − 1) 2 free parameters as the rows andcolumns must sum to the associated marginal probabilities.
This means that Σ [ j 1 ] , [ j 2 ] in the CLG 116 model is not identifiable.
We develop an identifiable variant of the CLG model in the supplement thatuses only ( K − 1) 2 parameters in Σ [ j 1 ] , [ j 2 ] .
Both models share similar imputation performance, butthe identifiable model is not invariant under permutation of the categorical labels while the model in
Definition 1 is invariant.
For the rest of this paper, we use the permutation-invariant CLG model.
2.1.3 Mixed categorical and ordered vector
We model mixed data that contains both categorical and ordered variables by combining the CLGmodel for categorical variables with the Gaussian copula model for ordered variables [9, 32].
To model x with ordered variables, the Gaussian copula assumes x ∼ GC (Σ , f ) is generated asan elementwise transformed Gaussian, i.e., x = f ( z ) =
( f 1 ( z 1 ) , . . .
, f p ( z p )) , where each f j is amonotonic increasing function and z ∼ N ( 0 , Σ) .
Denoting the cumulative distribution function(CDF) of x j as F j , the transformation f j is uniquely determined as f j = F − 1 j ◦ Φ , where Φ denotesthe standard Gaussian CDF and F − 1 j ( y ) :
= inf { x ∈ R : F j ( x ) ≥ y } .
Specifically, ordinals resultfrom thresholding the latent Gaussian variable.
For an ordinal x j , the transformation f j ( z ) has the form (cid:80) s ∈ S 1 ( z > s ) where 1 ( z > s ) is
if z > s and 0 otherwise.
Here, the set S is determined by
F j .
The CLG model has the same form as x = f ( z ) by writing
[Equation]
Hence, we propose the extended Gaussian copula (EGC) model.
Definition 2 (Extended Gaussian copula) .
Write a mixed data vector as x = ( x cat , x ord ) wherex cat collects all categorical variables and x ord collects all ordered variables.
We say x followsthe extended Gaussian copula x ∼ EGC (Σ , f ord , µ ) if there exists a correlation matrix Σ , anelementwise monotone f ord , and a µ such that x cat ∼ CLG (Σ cat, cat , µ ) and x ord ∼ GC (Σ ord,ord , f ord ) ,where Σ = (cid:20) Σ cat, cat Σ cat, ord Σ ord, cat Σ ord,ord (cid:21) .
We can also write x = f ( z ) =
( f cat ( z cat ; µ ) , f ord ( z ord )) for z = ( z cat , z ord ) ∼ N ( 0 , Σ) where f isdefined by all transformation parameters, i.e., f ord and µ .
Unlike the original Gaussian copula, thedimension of the latent z does not match that of the data x ; instead, it is larger as subvectors of zcorrespond to categorical entries of x .
The correlation matrix Σ must also obey constraints on thesubmatrix Σ cat, cat according to Definition 1.
2.2 Missing data imputation
Now we show how to impute missing values given an EGC model with known parameters.
For modelparameter estimation, see Section 2.3.
Note each categorical x j corresponds to K consecutive entries in z : the subvector z
[ j ] that generatesx j .
We use [ I ] to denote the set of indices in z that corresponds to a set of indices I from x .
Forexample, [ j ] follows this notation with I = { j } .
Two sets of indices are particularly important: theobserved entries O and missing entries M for a given mixed data vector x .
Thus [ O ] and [ M ] arethe latent dimensions in z that generate x O and x M , respectively.
Our imputation strategy follows [32].
First, the algorithm identifies a region of the latent space z
[ O ]constrained by the observed entries.
Then, it imputes the latent dimensions by exploiting the Gaussiandistribution of z [ M ] conditional on z [ O ] .
Finally, the algorithm uses the marginal transformation f toproduce imputations in the data space x M .
Pictorially, we can visualize this process as
[Equation]
Multiple imputation.
Multiple imputation creates several imputed datasets by sampling from thedistribution of missing entries conditional on the observations.
For the EGC model, Algorithm 1samples from the distribution of x M using two facts: (1) given x O , the variable z
[ O ] follows atruncated Gaussian distribution truncated to f − 1 O
( x O )
= (cid:81) j ∈O f − 1 j ( x j ) ; and (2) the random variablez [ M ] | z
[ O ] is Gaussian distributed.
Section 2.3.3 shows how to sample from the truncated Gaussianrandom variable given by z [ O ] | x O .
Using the empirical distribution of drawn samples, we canestimate the category probability for a missing categorical variable and build confidence intervals fora missing continuous variable.
O ) 4:
Sample ˆ z ( s )
[ M ] ∼ z
[ M ] | z
[ O ] :
N (cid:16) Σ [ M ] , [ O ] ˆ z ( s )
[ O ] , Σ [ M ] , [ M ] − Σ [ M ] , [ O ] Σ − 1 [ O ] , [ O ] Σ
[ O ] , [ M ] (cid:17) 5:
Compute ˆ x ( s ) M = f M (ˆ z ( s )
[ M ] ) 6: end for 7: Output: { ˆ x ( s ) M | s = 1 , .., m } .
Single imputation.
To provide an accurate single imputation under the EGC model, we imputebased on the conditional mean of the latent Gaussian, i.e.,
[Equation]
To compute E [ z [ O ] | x O , Σ , f ] , see Section 2.3.3.
With a known mean E [ z [ O ] | x O , Σ , f ] and marginaldistribution f M , computing the imputation ˆ x M is straightforward.
Specifically, for categorical missing x j , we first compute the K -dim
ˆ z
[ j ] :=
E [ z [ j ] | x O , Σ , f ] and then impute as ˆ x j =argmax(ˆ z [ j ] + µ [ j ] ) .
Online imputation.
Online imputation is the task to impute missing entries in data streams arrivingat different times.
We can conduct online imputation using the EGC model following [30]: uponobserving an incomplete data batch, impute the missing entries with a current saved model, and thenupdate the model using a new observation.
See the supplement on how to update the model.
2.3 Parameter estimation
Parameter estimation for the EGC model consists of two steps: (1) we form an estimate ˆ f of themarginal transformation f , and (2) we estimate the copula correlation Σ given ˆ f .
The marginaltransformation differs for ordered and categorical variables.
For an ordered x j , since f j = F − 1 j ◦ Φ ,we can simply estimate F − 1 j using the empirical quantile function on the observed entries of x j as in[19, 32].
For a categorical variable x j , we show how to estimate f j ( · ; µ [ j ] ) in Section 2.3.1 and howto estimate Σ in Eq.
(3) by extending estimation algorithms from [32].
2.3.1 Marginal estimation for categorical variables
Here we estimate the mean µ for which the EGC model, EGC (Σ , f ord , µ ) , matches the observedcategorical marginal distribution.
Each subvector µ [ j ] corresponding to categorical x
j may beestimated independently of all the others, so we drop the variable subscript j to ease the notation.
We must find µ in Eq.
(2) so that the resulting category frequency p k matches the observed frequency,subject to µ 1 = 0 : this setup gives K − 1 nonlinear equations with K − 1 unknowns.
Unfortunately,there is no closed form solution.
Thus we resort to iterative root finding algorithms, which use asoracles the value and derivative of P [argmax( z + µ )
=
k ] in Eq.
We use two approximationtechniques to evaluate these oracles efficiently.
First we approximate the probability of argmax as the expection of softmax by noticing
[Equation] [Equation]
where softmax( z k ; z ) = exp( z k ) / (cid:80) Kl
=1 exp( z l ) .
This approximation is accurate for large β .
Secondwe approximate the expectation using Monte Carlo samples with the reparameterization trick [18]:
[Equation]
where ¯ z i i.i.d.
∼ N ( 0 , I K ) .
With these approximations, we solve for µ subject to µ 1 = 0 in Eq.
[Equation]
[Equation] [Equation]
for k = 2 , . . .
, K .
The modified Powell method implemented in SciPy computes
2.3.2 Copula correlation estimation
Now we show how to find the maximum likelihood estimator (MLE) of copula correlation Σ , whichmodels the multivariate dependence structure.
Given i.i.d.
samples x i iid ∼ EGC (Σ , f ord , µ ) observed at O i and missing at M i for i = 1 , . . . , n .
We maximize the observed likelihood by integrating overthe latent space that maps to the observation x
i O
i , as in Eq.
[Equation]
The form of this likelihood matches that of the Gaussian copula for ordered variables [32] except thatthe latent integration is in dimensions [ O i ] instead of O i .
Hence the Expectation Maximization (EM)algorithm in [32] also works in our case by integrating over the appropriate latent indices.
Concretely,for the E-step at iteration t + 1 , we first compute the expectation of the joint likelihood of ( z i , x i O i )over the distribution of z i conditional on the observation x i
O
i and the previous correlation estimateΣ ( t ) using the Gaussianity of z i , denoted as Q (Σ; Σ ( t ) ) :
[Equation]
where c is a constant.
Then we compute argmax Σ Q (Σ; Σ ( t ) ) in the M-step, which is the expected“empirical covariance matrix” of z i .
We update Σ ( t +1) to its corresponding correlation matrix tosatisfy the copula parameter constraint similar to [11, 32]:
[Equation]
where P cor (Σ) returns the correlation matrix corresponding to a covariance matrix Σ .
The EMalgorithm is guaranteed to strictly increase the likelihood in Eq.
(7) and converges to a stationarypoint [22, Chapter 3].
The main computation is the expected sample covariance of z i .
Dropping therow index, evaluating E [ zz ⊤ | x O , Σ ( t ) ] requires computing E
[ z [ O ] | x O , Σ ( t ) ] and Cov[ z [ O ] | x O , Σ ( t ) ] .
We give details in the supplement and show how to evaluate the reduced quantities in Section 2.3.3.
Recall the correlation Σ must satisfy the constraint in Definition 1: the submatrix Σ
[ j ] , [ j ] for eachcategorical index j must be the identity.
The correlation computed by EM generally does not satisfythis constraint.
Instead, we use the function P cat (Σ) to correct the correlation Σ :
[Equation]
A
[Equation]
diag
[Equation]
by recognizing that Cov[Σ − 1 / 2 [ j ] , [ j ] z [ j ] ] = I K for each categorical index j , when x cat has p cat entries andx ord has p ord entries.
categorical then
[ j ] as the solved solution ˆ µ .
7: else 8:
Compute ˆ f j = ˆ F − 1 j ◦ Φ where ˆ F − 1 j is the empirical quantile function of X j .
9: end if 10: end for ▷ Marginal estimation ends 11: for t = 0 , 1 , 2 , . . .
until convergence do ▷ EM algorithm starts 12:
Compute E
[ z i ( z i ) ⊤ | x
i O i , Σ ( t ) , ˆ f ] for i = 1 , . . . , n .
Update Σ ( t +1) as in Eq.
Correct Σ ( t +1) ← P cat (Σ ( t +1) ) as in Eq.
15: end for ▷ EM algorithm ends 16:
Output: marginal estimation ˆ f , correlation estimation ˆΣ = Σ ( t +1) .
Computational cost.
Algorithm 2 summarizes the complete estimation algorithm for the EGCmodel.
Marginal estimation only takes a small fraction of total runtime (less than 4% in all experiments of this paper).
We observe that the EM algorithm mostly converges in less than 50 iterations.
Each iteration has time complexity O ( αnd 3 ) where α denotes the observed entry ratio, n denotesthe number of samples and d = p ord + p cat K denotes the latent dimension.
Line 12 of Algorithm 2takes the vast majority of the computation time.
There are many ways to accelerate the computation.
For large n , we can parallelize the computation in line 12 of Algorithm 2 or conduct minibatch EMtraining as in [30].
The minibatch EM performs model updates using randomly sampled data batchand achieves significant speedup without sacrificing accuracy.
For large d , we can reduce the cubictime complexity in terms of d to linear by imposing a low rank structure on Σ following [31], that isΣ =
WW ⊤ + σ
I d where W ∈ R d × k with k ≪ d and σ 2 ∈ (0 , 1) .
See the supplement for details.
2.3.3 Truncated Gaussian with categorical variables
Finally, we show how to estimate moments and sample from z [ O ] | x O , as these operations are requiredfor imputation and correlation estimation.
Note z
[ O ] | x O follows a truncated Gaussian distributiontruncated into the region f − 1 O
x O ) .
For each ordered variable with index j ∈ O , f − 1 j ( x j ) is aninterval.
(For continuous x j , this interval is a single point.)
Thus without observed categoricals,f − 1 O ( x O ) is a Cartesian product of intervals.
We call this distribution the interval-truncated Gaussian ,for which accurate sampling [2] and effective moment estimation [32] methods have been developed.
The truncation region is no longer a Cartesian product of intervals with observed categoricals.
Fortunately, for any x O we can find an involution A x (i.e., A 2 x = I ) so that A x z [ O ] | x O follows aninterval-truncated Gaussian distribution.
Consequently,
[Equation]
Similarly, to sample z i iid ∼ z
[ O ] | x O , we can first sample ˜
z i iid ∼ A x z
[ O ] | x O
and then use z i
=
A x ˜ z i .
Toderive
A x , first consider that when a categorical variable x takes value k , by Eq.
(3) we have:f
[Equation] [Equation]
; µ
[Equation]
= { z
[Equation]
R K : argmax
[Equation]
=
[Equation]
+
[Equation]
<
z
[Equation]
+ µ
[Equation]
, for j ̸ =
[Equation]
Define ˜ z
= A k z and ˜ µ =
A k µ where A k = − I K + (cid:80)
Ki
=1 E ik + E kk
(matrix E ij has 1 at ( i, j ) -thentry and 0 elsewhere).
We can rewrite Eq.
(12) to { ˜ z : ˜
z j + ˜ µ j > 0 , for j ̸ = k } , a Cartesian productof intervals.
We can find a matrix A j as shown above for every categorical variable x j .
Let A x
bediag
A 1 , . . . ,
A p cat , I p ord ) .
Then A z [ O ] follows interval truncated Gaussian and A x is an involution.
Experiments
In the experiments, we compare the accuracy of single imputations using our method EGC and severalcompetitors.
It is more challenging to compare the accuracy for multiple imputation (MI) as MI seeksto recover the correct distribution which is mostly unknown in practice.
As shown in Algorithm 2,
EGC does not require any model hyperparameter but only optimization hyperparamters M and β ,for which we use fixed values M = 5000 and β = 1000 across all our experiments.
We show in
Section 3.3 that this option already accurately estimates the categorical marginals.
Competitors.
We implement several imputation methods for mixed categorical and ordered data.
Among iterative imputation algorithms, we implement missForest [26, 25] and MICE [4, 28].
Among LRMC algorithms, we implement imputeFAMD [1, 17] and softImpute [20, 12].
Bothmethods one-hot encodes categorical variables.
imputeFAMD includes special weighting strategyincorporating different variables types, while softImpute simply treats the data as numerical.
We also implement a baseline imputation: majority vote for categorical variables and medianimputation for ordered variables.
We only need to choose hyperparameters for imputeFAMD andsoftImpute to have satisfying performance, for which we further mask 20% of observed trainingentries as a validation set.
Through all experiments in this paper, softImpute is the fastest methodand EGC comes second overall.
We include all implementation and runtime details in the supplement.
Evaluation.
To evaluate an imputation method, we compute misclassification error (ME) forcategorical variables and mean absolute error (MAE) for ordered variables at masked entries.
5 Number of categoricals 0.
M E
Imputation of categorical  5 Number of categoricals 1.
M A E
Imputation of continuous  5 Number of categoricals 0.
M A E
Imputation of ordinal
EGC_our missForest
MICE imputeFAMD softImpute baseline
3.1 Synthetic experiments
We generate synthetic datasets with 2000 samples and p ∈ { 11 , 13 , 15 } variables from the EGCmodel in Definition 2 with randomly drawn parameters.
The p variables include 5 exponentiallydistributed continuous variables, 5 ordinal variables with five ordinal levels, and p cat ∈ { 1 , 3 , 5 }categorical variables with six categories ( K = 6 ).
We randomly remove 30% entries of X forimputation evaluation and generate 10 masked datasets using different seeds.
Fig.
2 shows our method EGC performs the best for all variable types in all scenarios.
For categoricalvariables, EGC yields a larger improvement with fewer categorical variables.
For both continuousand ordinal variables, EGC substantially outperforms all other methods.
Several methods that imputecategorical variables quite well struggle to outperform a simple baseline on ordered variables.
Thesupplement includes more experiments that differ by the number of categories for categorical variables( K = 3 , 9 ), the missing ratio ( 20% and 40% ) and missing mechanisms (MAR and MNAR).
In general,these experiments show EGC performs well for both categorical and ordered variables in mixed datawhen the model is well-specified.
3.2 Real data experiments
We employ six real-world datasets from the UCI machine learning repository here [6].
Each datasetcontains both categorical and ordered variables (details in the supplement).
We randomly remove20% of observed entries as a test set to evaluate imputation accuracy and generate 10 masked datasetsusing different seeds.
Since variables in real data often have very different scales, e.g., numberof categories for categorical variables and standard deviation for continuous variables, we use anevaluation metric that compensates for the imputation difficulty across variables.
For an imputationmethod, we normalize its imputation error for each variable by the imputation error of baselineand then compute the average normalized error for each categorical variable and ordered variable,similar to [32].
We denote the scaled imputation error for categorical variables and ordered variablesas SME and SMAE, respectively.
Fig.
2 shows that our method EGC is one of the best method in almost all evaluation metrics.missForest comes second but its total runtime is four times longer than that of EGC .
Interestingly, softImpute imputes quite accurately for categorical data but poorly for ordered data.
Thesupplement includes more experiments that differ by the missing ratio and missing mechanism,demonstrating that EGC consistently outperforms other methods.
3.3 Categorical marginal estimation accuracy
Here we verify that EGC can learn arbitrary categorical distribution, i.e. the algorithm in Section 2.3.1finds the root µ of Eq.
(2) accurately.
Concretely, we compare two sides of Eq.
(2), where the rightside is the empirical category probability and the left side is approximated at the estimated
µ using10000 random samples.
Across all used datasets in Section 3.1 and Section 3.2, Fig.
3 shows the
[Figure] [Figure]
[Figure] [Figure]
estimated probability indeed matches the true probability well and the performance is robust to thenumber of categories.
Using a larger M in Eq.
(6) can further reduce the estimation error, but we findit does not significantly improve the imputation performance.
Discussion and conclusion
We have presented a probabilistic model, the EGC model, for mixed categorical and ordered data.
Wedeveloped an imputation method based on the EGC model that empirically demonstrates state-of-theart performance.
Here we discuss two limitations of this paper.
First, the estimation algorithm for the
EGC model admits guarantees under MCAR mechanism.
While it works reasonably well empiricallyunder other missing mechanisms, it may be possible to improve performance by estimating andexploiting the missing mechanism for model estimation.
Second, the EGC model uses a Gaussiandependence structure after a marginal transformation.
This paper demonstrates the benefits ofthe explicit marginal modeling.
Learning a more complex dependence structure, e.g., with neuralnetworks, may yield further improvements.
Checklist
For all authors...(
a ) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope
[Yes] (b) Did you describe the limitations of your work
[Yes] See Section {+
+}
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See the supplement
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] Provided in the supplemental material
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
the
[Table]
[Table]
[Table]
[Table]
[Table]
See
[Table]
If you are using existing assets (e
[Table]
g
[Table]
, code, data, models) or curating/releasing new assets..
[Table]
If your work uses existing assets, did you cite the creators?
[Yes
[Table]
) Did you mention the license of the assets?
[No] No license provided
[Table] ) Did you include any new [Table] either in the supplemental
[Table]
or as a URL?
[No]
[Table]
d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using [Table]
[No] All data is publicly available [Table]
) Did you discuss whether the data you are using/curating contains personally identifi- [Table]
able information or offensive content?
[No] The used data does not contain any ID [Table]
information or offensive content
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A] Research does not involve human subjects
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A] Research does not involve human subjects(c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A] Research does not involve human subjects
A An experiment on multiple imputation
Multiple imputation (MI) is often used in specific case studies ([14] e.g.) where no ground truth ispresent.
There is no generally accepted metric to compare MI methods.
Nevertheless, we designed asynthetic experiment scenario to showcase that our MI provides more accurate distribution estimationof the missing entries than MICE using much less time, and MICE is the only other imputationmethod implemented in this paper that supports MI.
For the designed synthetic experiment, the truedistribution of missing entries can be accurately approximated.
Thus we can compare the estimateddistribution provided from MI to the true distribution.
Specifically, we use a synthetic dataset with 5 exponentially distributed continuous variables and1 categorical variable with 6 categories, 2000 samples and 30% missing ratio, generated similar tothat in Section 3.1.
For straightforward evaluation, we focus on the categorical variable, for whichwe use its conditional distribution given all 5 continuous observations as the true distribution.
Weevaluate the distribution estimation error through the cross entropy loss (smaller is better) betweentrue and estimated category probability, averaged across all missing categorical entries.
For bothour EGC method and MICE, we take 100 multiple imputation samples and compute the empiricalprobability for each category as an estimate.
The results are reported in Table 1.
Our EGC MIachieves significantly smaller loss and smaller runtime than MICE.