V ARIATIONAL I NFERENCE FOR D ISCRIMINATIVE L EARNING WITH G ENERATIVE M ODELING OF F EATURE I NCOMPLETION
Anonymous authors
Paper under double-blind review
A BSTRACT
We are concerned with the problem of distributional prediction with incomplete features: The goal is to estimate the distribution of target variables given feature vectors with some of the elements missing.
A typical approach to this problem is to perform missing-value imputation and regression, simultaneously or sequentially, which we call the generative approach.
Another approach is to perform regression after appropriately encoding missing values into the feature, which we call the discriminative approach.
In comparison, the generative approach is more robust to the feature corruption while the discriminative approach is more favorable to maximize the performance of prediction.
In this study, we propose a hybrid method to take the best of both worlds.
Our method utilizes the black-box variational inference framework so that it can be applied to a wide variety of modern machine learning models, including the variational autoencoders.
We also confirmed the effectiveness of the proposed method empirically.
I NTRODUCTION
We address the problem of prediction with incomplete data, which often arises in real-life data due to the lack of data collecting resources and/or privacy concerns.
For example, consider analyzing electronic health records.
The input variables of the prediction are patients’ demographic characteristics and history of medical measurement data obtained with various instruments and the target variable is the survival time of patients.
Thus, some of these features are not available depending on patients due to, e.g., non-standardized medical equipment, legal regulation and privacy concerns.
More formally, each input is an incomplete set of features represented with a tuple u = (˜ x , m ) such that ˜ x ∈ R d x is the corrupted version of the complete input x ∈ R d x and m ∈ { 0 , 1 } d x indicates the missing entries of ˜ x
: For all j ∈ [ d x ] , m j = 1 if and only if the j -th feature is missing.
In particular, missing entries are filled with zero and non-missing entries are identical to the one in the complete input, ˜ x
j = (1 − m j ) x j .
Given the incomplete features u , we want to predict the outcome y ∈ R d y , which is modeled with a predictive distribution y ∼ p ( y | u , θ ) .
Here, the parameter θ is unknown and should be learned from data.
One straightforward way to deal with such incompletion is to incorporate generative models: The missing values in ˜ x are imputed with some generative model and the target y is predicted based on the result of the imputation.
Sometimes the imputation is done implicitly or simultaneously with the prediction.
An advantage of this method lies in that the meaning of m is inherently incorporated.
Without such context, algorithms must learn by themselves that m is actually indicating missingness (if that is relevant to the prediction task), which incurs extra generalization errors.
In other words, appropriate generative models make the prediction robust to the feature incompletion.
On the other hand, the discriminative models that directly learn the conditional density p θ ( y | u ) sometimes exhibit superior performance (Kuhn et al., 2013) due to their alignment in terms of the objective.
This is especially the case when the feature incompletion is not a dominant factor (Lasserre et al., 2006).
However, it is not as straightforward as generative models to incorporate the context of missingness into discriminative models and therefore their prediction may be less robust to the feature incompletion.
To best of our knowledge, there have been no general solution to this dilemma of the generative and the discriminative approaches, i.e., generic method of discriminative inference with incomplete features for generative models , which we abbreviate as DIG .
In particular, although there have been similar efforts made in the literature, the previous methods are limited in terms of the applicable models.
For example, several DIG methods have been proposed for exponential families (Ghahramani & Jordan, 1994; Smola et al., 2005) and Gaussian processes (Pacheco et al., 2014), but they cannot handle modern machine-learning architectures such as deep neural networks.
This motivates us to seek for general and widely applicable frameworks that solve the dilemma.
To this end, we propose a new variational approximation and a new variance reduction technique, which enable us to perform DIG within the black-box variational inference (VI) framework (Jordan et al., 1999; Zhang et al., 2018).
We have also empirically confirmed the effectiveness of the proposed method through numerical experiments, employing the variational autoencoders (VAEs) (Kingma & Welling, 2013) as the base generative model.
The rest of the paper is organized as follows.
In Section 2, we introduce the mathematical notation and the technical challenges in DIG with flexible models.
Then, in Section 3, we present our method adopting VI in the DIG setting.
In Section 4, we show the experimental results that confirm the effectiveness of the proposed method.
In Section 5, we review and discuss the related work.
Finally, in Section 6, we conclude the paper with some remarks and future perspectives.
All the proofs for theoretical statements are given in the appendix.
P RELIMINARY
In this section, we introduce the mathematical formulation of the DIG method and then review some background of its technical challenges.
2. DIG F ORMULATION
Let i ∈ [ n ] denote the index of instances, where n is the number of data and [ n ] denotes the set of integers { 1 , 2 , . . . , n } .
Let y [ n ] :=
{ y i ∈ R d y } i ∈ [ n ] and x [ n ] := { x i ∈ R
d x } i ∈ [ n ] , be the sets of the target and the complete feature vectors, where d y and d x are the dimensionalities of vectors.
We assume we can only observe the set of incomplete feature vectors u [ n ] := { u i :=
(˜ x i , m i ) } i ∈ [ n ] instead of x [ n ] , where ˜ x i ∈ R d x is the corrupted version of x i and m i ∈ { 0 , 1 } d x is the indicator of corruption (‘mask vector’) for all i ∈ [ n ] .
Each mask vector m i indicates which entry of ˜ x
i is corrupted; We have ˜ x ij =
(1 − m ij ) x ij for all i ∈ [ n ] and all j ∈ [ d x ] .
Generative model.
Suppose that the generative process of the observables ( y [ n ] , u [ n ] ) is modeled with a parametrized family of instance-wise identical distributions.
That is, the joint probability density of ( y [ n ] , u [ n ] ) is factored as
[Equation]
where the function p ( y i , u i | θ ) on the RHS denotes the common density function across instances i ∈ [ n ] indexed with any parametric models θ (e.g., by neural networks).
Discriminative objective.
The goal of DIG is to maximize the conditional likelihood of the corrupted data,
[Equation]
Now, let L i y | u
( θ ) :
= ln p ( y i | u i , θ ) denote the i -th summand.
Following Bayes’ rule, each L i y | u ( θ ) is computed with a difference of log likelihoods
[Equation]
[Equation] where L i y , u ( θ ) : = ln p ( y i , u i | θ ) and L i u ( θ ) :
= ln (cid:82) p ( y i , u i | θ ) d y i . Since we employ gradient-based optimization methods, it suffices to consider (approximate) evaluation of the value and the gradient of the instance-wise objective L i y | u ( θ ) if there is no risk of confusion.
2. DIG WITH L ATENT V ARIABLE M ODELS (DIG-LVM)
We give further details of DIG with a specific class of generative models, namely the latent variable models (LVMs).
LVM is one of the most common and expressive generative models.
In the context of DIG, the generative density function is given by
[Equation]
where z is the instance-wise latent variable, which is expected to capture some higher-order information of the observables.
For example, the variational autoencoders (VAEs) (Kingma & Welling, 2013), the generative adversarial networks (GANs) (Goodfellow et al., 2014) and the normalizing flow models (Rezende & Mohamed, 2015) are instances of LVM.
Modeling feature-corruption process with LVM. One advantage of LVM is that complex generative processes can be modeled with relatively simple joint density p ( y , u , z | θ ) .
This is especially beneficial in modeling missing data: The full joint density p ( y , u , z | θ ) can be easily designed to reflect one’s belief on the corruption process such as MCAR and MNAR (Rubin, 1976) through the factorization of the density (e.g., see Collier et al. (2020)), while maintaining the expressibility of LVM. For example, the MCAR process can be modeled with
[Equation]
where p ( m ) is the true marginal density function of m , which is unknown but cancels out in the DIG objective (Equation 3), and p ( z ) is an arbitrary fixed prior of the latent variable (such as Gaussian).
The MNAR process can be modeled with [Equation] The modeling of the other missing-value processes is relegated to the appendix (Section G) as it requires additional assumptions on the causal relationship between variables (Rubin, 1976).
Under these factorizations, the log likelihoods on the RHS of Equation 3 for LVMs are computed as
[Equation]
where both density functions in the integrals can be analytically computed under either of Equationand Equation 6.
Challenge: Inference with DIG-LVM.
To maximize Equation 3 with LVM, we have to evaluate the values and gradients of the likelihoods in Equation 7.
If there are only positive log integrals in the objective, this can be approximately done with the variational inference (VI) framework (Jordan et al., 1999; Zhang et al., 2018) in favor of flexibility and scalability.
However, with DIG, we also have a negative log integral, i.e., −L u ( θ ) .
The optimization of such objective is relatively difficult since there have been no equivalent of VI in terms of the flexibility and scalability (see Section 5).
V ARIATIONAL I NFERENCE FOR DIG-LVM
Now we present the main result, the method of VI for DIG-LVM.
In Section 3.1, we derive a variational approximation to the conditional likelihood given by Equation 3.
Then, in Section 3.2, we derive a method for optimizing the variational approximation.
Finally, in Section 3.4, we show the prediction procedure based on the optimized parameters.
3. V ARIATIONAL L OWER /U PPER B OUNDS
In this section, we discuss an approximation of L y , u ( θ ) and L u ( θ ) in Equation 7 individually.
Note that both are log integrals of the specific form
[Equation]
[Equation] where v represents either ( y , u ) or u .
Since such integrals constitute the objective L ( θ ) with positive and negative signs, our goal is to derive both upper and lower bounds on Equation 8.
Evidence Lower Bound (ELBO).
The lower bound is given in the standard way (Jordan et al., 1999; Zhang et al., 2018),
[Equation]
where ϕ is a variational parameter of a probability density function q ( z | v , ϕ ) and D KL
[Equation] E z
[Equation] z | v ,ϕ ) [ln q ( z | v ,ϕ ) p ( z | v ,θ ) ] is the KL divergence of the parameters ϕ
and θ , We call
L ELBO( v ) ( θ, ϕ ) as the evidence lower bound (ELBO) .
Since ELBO is an expectation of a tractable function, we approximate it with Monte-Carlo sampling.
Evidence Upper Bound (EUBO).
To derive an upper bound, we start with applying the χevidence upper bound (CUBO)
(Dieng et al., 2017).
For any real numbers α > 1 , CUBO is derived as follows:
[Equation]
where ψ is a variational parameter of a probability density function q ( z | v , ψ ) and
[-= 
-][Equation]
d z p α ( z | v , θ ) q 1 − α ( z | v , ψ ) denotes the α -R ´ enyi divergence of the parameters θ and ψ .
Note here, unlike ELBO, CUBO is not unbiasedly approximated because of the logarithm wrapping the expectation.
To address this issue, we apply another variational approximation with a divergence function Ψ α ( t ) :=(
e αt − αt − 1) /α , t ∈ R : Since Ψ α ( t ) ≥ 0 for all t ∈ R , we have
[Equation]
where ξ is a variational parameter of a real-valued function f ( v ; ξ ) .
We call the right-hand side as the evidence upper bound (EUBO) .
Note that the expectation in EUBO is linearized and thus can be unbiasedly approximated with Monte-Carlo estimation.
Conditional Evidence Lower Bound (CELBO).
Applying ELBO on v = ( y , u ) and EUBO on v = u , we have a conditional evidence lower bound (CELBO),
[Equation]
By definition, CELBO bounds the DIG objective from below, L y | u ( θ ) ≥ L CELBO ( y | u ) ( θ, ϕ, ψ, ξ ) .
The inequality is tight for any generative parameter θ owing to the tightness of ELBO and EUBO, 1 i.e., for all θ , there exists a tuple of variational parameters ( ϕ, ψ, ξ ) such that the gap ∆( θ, ϕ, ψ, ξ ) :
= L y | u ( θ ) − L CELBO ( y | u ) ( θ, ϕ, ψ, ξ ) is zero.
Moreover, CELBO can be unbiasedly approximated as well as ELBO and EUBO.
3. O PTIMIZATION A LGORITHM : V DIG
Since CELBO can be unbiasedly approximated
[Equation]
we may employ stochastic gradient-based optimization to maximize it.
The full stochastic objective for CELBO is given by
[Equation]
where z ϕ and z ψ are Monte-Carlo samples drawn from q ( z | y , u , ϕ ) and q ( z | u , ψ ) , respectively.
The gradients of ˆ L CELBO ( y | u ) is taken with any standard automatic differentiation libraries, using the reparametrization trick (Kingma & Welling, 2013) or the REINFORCE trick (Williams, 1992).
Since the actual objective function is the summation of individual losses ˆ L CELBO ( y | u ) = ˆ L i CELBO ( y | u ) over all the instances, we may draw a minibatch of instances for each iteration.
We call the resulting inference algorithm as vDIG (Algorithm 1).
3. V ARIANCE R EDUCTION FOR V DIG WITH S URROGATE P ARAMETRIZATION (SP)
The boundedness of the norm of the stochastic gradient ∇ ˆ L CELBO ( y | u ) ( θ, ϕ, ψ, ξ ) is crucial for the stable and fast convergence of stochastic gradient-based algorithms like Algorithm 1.
However, the stochastic CELBO contains the density ratio
[Equation]
[Equation] raised to the power of α , which is problematic as the ratio w θ, ψ,ξ may have large variance and so does the gradient.
The key idea is to regularize the parameter to keep the density ratio small.
Note that we have w θ,ψ,ξ ( · , · ) ≡ 1 whenever the objective gap is zero.
That is, constraining the parameter to satisfy sup u , z w θ,ψ,ξ ( u , z ) ≤ 1 does not lose the model expressibility if the variational approximation is tight.
The problem is, enforcing such constraint during optimization is intractable in general.
We address this issue by introducing a surrogate parametrization.
Define new parameters θ ′ and ξ ′ formally 2 by
[Equation]
where G ( w ) :=(1 ∨ w ) − 1 { 1 + α ln(1 ∨ w ) } 1 /α ( w ≥ 0) , a ∨ b := max { a, b } , and Z ( θ ′ ) is the normalizing constant ensuring the mass preservation of the density under θ ′ .
We refer to the mapping
T SP : ( θ, ϕ, ψ, ξ ) (cid:55)→ ( θ ′ , ϕ, ψ, ξ ′ ) as the surrogate transform, and G ( w ) as the gain function.
The surrogate transform T SP and the gain function G ( w ) are designed carefully to satisfy the following two properties.
First, it preserves the effective parameters of CELBO.
Definition 1 (Effective parameters) .
Let Ω be a set of CELBO parameters ( θ, ϕ, ψ, ξ ) .
Then, we define the effective parameters of Ω by Θ 0 (Ω) :
= { ( θ, ϕ, ψ, ξ ) ∈ Ω : ∆( θ, ϕ, ψ, ξ ) = 0 } , i.e., the set of parameters inducing tight variational approximation.
Proposition 2 (Effective parameter preservation) .
For arbitrary Ω , we have Θ
( T SP (Ω))
=
Θ 0 (Ω) .
In other words, in a sense, T SP does not alter the original model.
See Section H for extended discussion.
Second, it regularizes the growth of the highly stochastic term (Equation 14).
Intuitively, the effect of regularization is quantified with the gain function G ( w ) since it represents the ratio of stochastic term before and after the transform,
[Equation]
Note that G ( w ) is no larger than one and non-increasing for w ≥ 0 , thus the stochasticity is reduced through the transform.
See Figure 3 in the appendix for visualization.
Consequently, T SP guarantees the boundedness of the stochastic gradient.
Proposition 3 (Bounded gradient) .
Assume there exists G > 0 such that ∥∇ ln p ( y , u , z ϕ | θ ) ∥ , ∥∇ [Equation] p [Equation]
u ,
[Equation] ψ [Equation] , ∥∇ ln
[Equation] [Equation] | y , u , ϕ ) ∥ , ∥∇ ln q ( z ψ | u , ψ ) ∥ , ∥∇ f ( u ; ξ ) ∥ ≤
G .
Then, ∥∇ ( ˆ L CELBO ( y | u ) ◦ T SP ) ( θ, ϕ, ψ, ξ ) ∥ ≤ 9 G .
Proposition 2 and 3 justifies optimizing the objective via the transform T SP ; we get a gradientnorm bound without altering the effective parameters.
The full objective function after the surrogate transform, ˆ L CELBO-SP ( y | u ) :
= ˆ L CELBO ( y | u ) ◦ T SP , is given by [Equation]
[Equation]
which we call the stochastic CELBO-SP.
The key point is that computing the stochastic CELBO-SP does not require the computation of the normalizing constant Z ( θ ′ ) , which is intractable in general.
This is not the case with the surrogate transformation on ELBO or EUBO alone.
The CELBO-SP maximization is done by simply replacing CELBO with CELBO-SP in Algorithm 1.
3. P REDICTION A LGORITHM
Given ( θ, ϕ, ψ, ξ ) trained by Algorithm 1, we want to compute the predictive distribution on new instances given their incomplete features u new := (˜ x new , m new ) .
Since the conditional density p θ ( y | u ) is intractable in general, we approximate it with the Monte-Carlo method.
The approximated conditional distribution is given by
[Equation]
where z sψ are independently drawn from q ψ ( z | u ) , s ∈
[ k pred ] .
This procedure is justified as follows.
Proposition 4.
Let p ( y | u , ¯ θ ) :
=
E [ p ( y | u , ˆ θ )] , where the expectation is taken with respect to the
Monte-Carlo sampling.
Then,
[Equation]
In other words, if the objective gap is small, so is the approximation error of ¯ θ , which is the limit of the actual predictor ˆ θ with k pred → ∞ .
In the experiment, we used k pred = 512 .
E XPERIMENTS
In this section, we demonstrate the effectiveness of the proposed method, i.e., Algorithm 1, through numerical experiments.
We first introduce a number of methods compared in the experiments in
Section 4.1.
We then present the procedures and results of three experiments designed to show i) the effectiveness of the new variational approximation, ii) the superior performance of the proposed method, and iii) the robustness of the proposed method against feature corruption.
4. E XPERIMENTAL S UBJECTS
We employ three subject algorithms and two baselines in the experiment.
The main subjects are all based on VAE (Kingma & Welling, 2013), a typical example of LVM, whose basic architectures are identical to each other: See the appendix (Section D) for more details.
Generative method: VAE, VAE*.
As an example of generative approach, we employ VAE and solve missing-value imputation and regression simultaneously by regarding the target as a part of the feature, x ′ ← x ⊕ y , and reconstructing x ′ from its corrupted version, where the entries corresponding to y is considered as missing.
In particular, we adopt the formulation of Collier et al.
(2020), which comes with two variants for different missing-value processes, namely missing not at random (MNAR) and missing completely at random (MCAR).
The difference between MNAR and MCAR is only in the generative model (i.e., the architecture of the decoder), represented by Equation 6 and Equation 5, respectively.
We denote these variants as VAE and VAE*, respectively.
To train it to be able to reconstruct y , we double the training dataset with masking (and not masking) the target y .
Note that the objective function is not exactly aligned with the prediction task because of the imputation-based formulation.
Discriminative method: CVAE.
As an example of discriminative approach, we employ the conditional VAE (CVAE) proposed by Kingma et al.
(2014); Sohn et al.
In our setting, the conditional variables consist of the corrupted feature ˜ x and the missingness indicator m .
In practice, these vectors are concatenated before fed into neural networks.
The generative model of CVAE in this setting is given by [Equation] while the decoder represents the density in the integral and the encoder is used to approximate the posterior p ( z | y , u , θ ) . Note that the resulting model is not informed with the context of m as VAE and VAE* are, 3 but the objective is (a variational approximation of) the conditional evidence, which is aligned with the prediction task.
Proposed method: DVAE, DVAE*.
We call the instantiation of vDIG on VAE as the discriminative VAE (DVAE).
We implement MNAR (Equation 6) and MCAR (Equation 5) variants of DVAE as with VAE, respectively denoted by DVAE and DVAE*.
The difference with VAE and its variant is that we have additional encoders corresponding to ψ and ξ , while the decoder θ and the encoder ϕ are common.
See Section D for more details.
We choose α = 2 for the parameter of the R ´ enyi divergence.
Baseline methods: Simple, MICE.
As the first baseline method, we employ a simple fullyconnected feed-forward neural network (FCFNN) with the same architecture with the encoders of the above methods, except its output is considered as a distribution of the target y instead of z .
Although the output distribution is restricted to a Gaussian, it requires no variational approximation and hence the conditional likelihood ln p θ ( y | u ) is exactly maximized.
We call this Simple.
The second baseline is the method of the multiple imputation by chained equations (MICE) (Azur et al., 2011), which is a classic approach in statistics to deal with missing values.
In particular, we employ the Bayesian ridge regression for the missing-value imputation task and the FCFNN of the first baseline for the subsequent regression task.
4. E XPERIMENTAL P ROCEDURE AND R ESULTS
The organization of the experiments is three-fold.
First, in Section 4.2.1, we check the effectiveness of our variational approximation techniques, namely EUBO and the surrogate parameterization.
Second, in Section 4.2.2, we compared the proposed method(s) with the existing methods in terms of the predictive performance.
Finally, in Section 4.2.3, we examine the robustness of these methods against the change in the missing-value ratio.
See also the appendix for the details of the experimental settings, including the information of the datasets.
4.2. E FFECT OF N EW V ARIATIONAL A PPROXIMATION
Procedure.
We compare the stability of optimization of DVAE with three different variational approximation, namely, ELBO+CUBO, ELBO+EUBO and ELBO+EUBO+SP.
In all cases, the
[Figure] [Figure]
Total
CVAE 0.41 (0.02) 5.19 (0.14) 5.33 (0.02) 3.01 (0.18) 5.46 (0.03) 3.49 (0.06) 3.82 (0.04) DVAE 0.45 (0.03) 5.16 (0.10) 5.26 (0.02) 2.91 (0.23) 5.44 (0.08) 3.36 (0.04) 3.76 (0.05) DVAE* 0.44 (0.03) 5.16 (0.11) 5.27 (0.03) 2.92 (0.21) 5.42 (0.07) 3.35 (0.02) 3.76 (
[Table]
Simple 0.49 (0.03) 5.47 (0.11) 5.40 (0.02) 3.13 (0.18) 5.47 (0.04) 3.62 (0.04) 3.93 (
[Table]
04) MICE 0.46 (0.06) 5.34 (
[Table]
[Table] [Table]
05) VAE 0.47 (0.01) 5.21 (0.08) 5.49 (0.06) 2.95 (0.23)
[Table]
Monte-Carlo approximation of ELBO is used to compute the joint evidence L y , u ( θ ) .
The marginal evidence L u ( θ ) is computed with the Monte-Carlo approximation of CUBO in ELBO+CUBO (albeit biased) and of EUBO in both ELBO+EUBO and ELBO+EUBO+SP.
Only ELBO+EUBO+SP employs the surrogate parametrization.
Result.
Figure 1 shows the training processes of DVAE with the AQ-CO dataset from UCI Machine Learning Repository (see the appendix).
It is seen that ELBO+EUBO+SP is most stable in the optimization.
Also note that the objective of ELBO+CUBO may be negatively biased (recall the logarithm wrapping the expectation in Equation 10), but we cannot know how much.
4.2. P REDICTIVE P ERFORMANCE
Procedure.
We apply seven different algorithms in Section 4.1 to six different regression tasks from UCI Machine Learning Repository, summarized in Table 3 (in the appendix).
We drop the entries of feature completely at random with probability p = 0 .
In each configuration, we iterate the same procedure with five different random seed values.
Result.
Table 1 shows the cross entropy, i.e., − ln p θ ( y | u ) , averaged over the test splits.
Overall, both DVAE and DVAE* almost always perform best or at least comparable to the best scores within one standard deviation, indicating the effectiveness the DIG method applied VAE.
In particular, it is more clear that they outperform the others when averaged over all the datasets (‘Total’ column).
Moreover, DVAE* is slightly better than DVAE as expected because the way we drop the feature entries is MCAR.
4.2. R OBUSTNESS AGAINST F EATURE C ORRUPTION
Procedure.
We take the result of Section 4.2.2 and examine how the missing-value ratio p affects predictive performance.
For each method, we calculate the difference between the average cross entropy with p = 0 .
1 and p = 0 .
Result.
The difference is visualized in Figure 2.
The most robust algorithms are VAE and VAE*, while DVAE* and DVAE are the second and the third most robust algorithms.
This matches our
[Figure]
D i ff e r e n c e i n
C r o ss E n t r o p y expectation because both VAE and VAE* are informed with the context of m and implicitly regularized through missing-value imputation task.
Similarly, DVAE and DVAE* are also informed with the meaning of m , but they are not subject to the implicit regularization.
In particular, DVAE* is more robust than DVAE as the MCAR model is accurate (in this setting) and more informative than the MNAR model.
R ELATED W ORK
In this section, we review existing variational approximations bounding the log-integrals from above .
We also discuss the relationship with the existing studies on missing data handling in the appendix.
The χ -variational upper bound (CUBO) (Dieng et al., 2017) has been proposed to estimate such upper bounds, utilizing the α -R´enyi divergence (equivalently, χ -divergence).
However, as Pradier et al.
(2019) pointed out, the resulting estimate of CUBO is biased and in some cases numerically unstable 4 .
This phenomenon has been also confirmed in our experiment (Section 4.2.1).
Another upper bound has been proposed by (Ji & Shen, 2019) with the reverse KL divergence instead of the R ´ enyi divergence, but it is also biased and not guaranteed to be an upper bound.
We also find that the parsimonious upper bound (Mattei & Frellsen, 2018) is suitable in terms of the upper-bound guarantee, while it induces a min-max form during the upper bound minimization procedure, which implies the convergence property of the algorithm could be tricky.
The proposed upper bound, EUBO, is closely related to the one studied by Kuleshov & Ermon (2017), which is designed for estimating partition functions.
Although their method also suffers from high variance, they took different approach to reduce it.
In particular, they adaptively reduced the learning rate of the variational parameter ψ .
However, as the authors noted, this does not solve the problem enough for scaling to large datasets.
C ONCLUSION
We have proposed a novel algorithm to perform discriminative training with incomplete features for generative models, which is derived on the basis of a newly introduced variational approximation and parameter transformation.
The effectiveness of the proposed method has been confirmed in terms of the stability of the new upper bound and the predictive performance and robustness of the resulting algorithm.
Possible directions of future work include the application of the surrogate transform technique to other contexts than feature incompletion.
A P ROOFS A. P ROOF OF P ROPOSITION
Proof.
Θ 0 ( T SP (Ω))
⊃ Θ 0 (Ω) is trivial as ∆( θ, ϕ, ψ, ξ ) = 0 implies w θ,ψ,ξ ( · , · ) ≡ 1 , which implies ( θ, ϕ, ψ, ξ ) is a fixed point of T SP , hence ( θ, ϕ, ψ, ξ ) ∈ T SP (Θ) .
The other direction is shown as follows.
Assume ∆( θ ′ , ϕ, ψ, ξ ′ )
= 0 , where ( θ ′ , ϕ, ψ, ξ ′ )
=
T SP ( θ, ϕ, ψ, ξ ) and ( θ, ϕ, ψ, ξ ) ∈ Ω .
Then, we have
[Equation]
for all u and z .
This implies w θ,ψ,ξ ( · , · ) ≡ 1 as wG ( w ) =
1 ⇔ w = 1 .
Therefore, ( θ, ϕ, ψ, ξ ) is a fixed point of T SP and ( θ ′ , ϕ, ψ, ξ ′ ) ∈ Ω .
A. P ROOF OF P ROPOSITION
Proof.
Let g :=
∇ ( ˆ L CELBO ( y | u ) ◦ T SP )( θ, ϕ, ψ, ξ ) and observe [Equation]
[Equation]
where w :=
w θ,ψ,ξ ( u , z ) , H ( t ) :
= ln G ( e t ) − ( e t G ( e t ))
α /α , and H ′ ( t ) denotes the derivative of
H ( t ) .
Thus, we have
[Equation]
as | H ′ ( t ) | ≤ 2 and ∥∇ ln w ∥ ≤ 3 G .
A. P ROOF OF T HEOREM
Proof.
According to the information processing inequality, we have D KL ( y | u ) ( θ ∥ ¯ θ ) ≤
D KL ( z | u ) ( θ ∥ ψ ) . Moreover, by the construction of L MEUBO , we have (1 − α − 1 ) D α
( z | u ) ( θ ∥ ψ ) ≤ ∆( θ, ϕ, ψ, ξ ) .
The desired result is seen by combining these two inequalities with the fact that the α -R ´ enyi divergence dominates the KL divergence for all α > 1 .
B E XPERIMENTAL C ONFIGURATIONS
Computing infrastructure.
The computing infrastructure used in the experiments is summarized in Table 2.
[Table]
Preprocessing.
All datasets used in the experiment are standardized before fed into algorithms so that the variables of the feature and the target have zero mean and unit empirical variance on the training split.
More precisely, we apply the following transformation
[Equation]
where ˆ µ x,j := (cid:80) ni
=1 (1 − m ij ) x ij (cid:80) ni
=1 (1 − m ij ) and ˆ σ ⊙ 2 x,j := (cid:80) ni
=1 (1 − m ij )( x ij − ˆ µ x,j ) ⊙ 2 (cid:80) ni
=1 (1 − m ij ) for j ∈ [ d x ] , and ˆ µ y := 1 n (cid:80) ni
=
y i and ˆ
σ ⊙ 2 y := 1 n (cid:80) ni
=
( y i − ˆ µ y ) ⊙ 2 .
Here, ⊙ and ⊘ denotes the elementwise multiplication and division of vectors, respectively.
Note that we refill the missing entries of ˜ x
i with zero after the affine transformation, which is corresponding to filling them with the means in the original dataset.
In the prediction step, the input feature ˜ x undergoes the same transformation, ˜ x i ← (1 − m ) ⊙ (˜ x − ˆ µ x )
⊘ ˆ σ y , and the predictive distribution p ( y | u , θ ) for the standardized target is made based on it.
Then, the distribution is transformed inversely to predict the non-standardized target, p ( y | u , θ ) ← ( (cid:81)
d y j =1 ˆ σ − 1 j,y ) · p (( y − ˆ µ y )
⊘ ˆ σ y | u , θ ) .
Validation split.
We randomly hold 20% of the training split out for validation.
Algorithms receive as the input only the remaining 80%.
Initialization.
All the parameters involved in the experiments are contained in the Linear module of Pytorch and initialized with its default initialization method, i.e., independently subject to the uniform distribution on the interval [ − 1 √ k , 1 √ k ] , where k is the number of input variables for the respective modules.
First-order optimizer.
In all the experiments, we employ the AdaBelief optimizer (Zhuang et al., 2020) with its default setting, 5 which is recently proposed as an improved alternative for the Adam optimizer (Kingma & Ba, 2014).
The size of minibatch is always taken to be 512.
In particular, for sampling-based methods (i.e., VAE, VAE*, CVAE, DVAE, DVAE*), the stochastic gradient is computed with the reparametrization trick.
Number of Iterations.
We iterate the loop 2000 times for each configuration, i.e., 2000 ×samples are seen in one run irrespective to the data size.
Evaluation.
In the above iteration, we save the models at 20 predefined points t ∈ { 0 , 1 , 2 , 3 , 5 , 8 , 13 , 19 , 29 , 43 , 64 , 94 , 138 , 203 , 298 , 436 , 638 , 934 , 1366 , 1998 } , which are (approximately) equally spaced in the log scale.
Finally, the best model among these 20 checkpoints is chosen with respect to the cross entropy on the validation split and evaluated on the test split.
If the subject algorithm is based on LVM, then we sample k pred = 512 instances from the posterior distribution q ( z | u , ψ ) and make prediction with Equation 16.
C D ATASETS
All the datasets used in the experiments are taken from UCI Machine Learning Repository.
AQ-CO,
AQ-NMHC and AQ-NOx are taken from the AirQuality dataset (De Vito et al., 2008) and corresponding to different targets denoted by their suffixes.
YearPred is a part of the YearPredicitonMSD dataset (Bertin-Mahieux, 2011), where, to accommodates the fast iteration of the experiments, the number of records is restricted to 10,000 by separate random sampling for training (8000 records) and test (2000 records) splits.
Boston and Diabetes are respectively taken from the dataset of the same names.
The basic statistics are summarized in Table 3.
[Table]
D M ETHODS C OMPARED
In this section, we show the detailed architecture of the methods compared in the experiments.
Both the encoder and the decoder are fully-connected feed-forward neural networks (FCFNN) with single hidden layer of width 256 and ReLU activation that take a vector input t ∈ R k , k ≥ 1 , and emit a diagonal Gaussian distribution depending on t .
To facilitate the specification of the architectures, we first define some basic building blocks.
Let every distinct occurrence of Linear d ( · ) denote a distinct linear layer with the output dimension d ≥ 1 with its own parameters ( W, b ) such that Linear d ( t )
=
W t + b with a weight matrix W ∈ R d ×∗ and an offset vector b ∈ R d for t ∈ R ∗ .
The Gaussian layer is defined by the composition of the Gaussian density function and the linear layers,
[Equation]
where ϵ := 10 − 8 is a small constant and N d [ t | v , s ] denotes the d -product of the probability density functions of the Gaussian distributions with mean v j and standard deviation s j evaluated at t j , j ∈ [ d ] , for t , v ∈ R d and s ∈ R d> 0 .
We also denote the ReLU activation function by ReLU ( t ) :
= max { 0 , t } .
Finally, we define the composition of a Gaussian layer and a Linear layer with k hidden neurons and ReLU activation function by
[Equation]
Now, we are ready to describe the architecture specifications.
Simple.
The objective of the Simple baseline is given by
[Equation]
where θ denotes all the parameter implicitly involved in the RHS.
MICE.
MICE is implemented with IterativeImputer from scikit-learn (ver. 0.24.2) with its default argument except with the burn-in period changed from 10 to 40.
After the imputation, only the final imputation result is passed to the Simple baseline to reduce the computation time.
VAE, VAE*.
The objective of VAE is given slightly modifying Collier et al.
The difference is that the mask vector m is modeled as a conditional variable rather than a random variable.
The reason of this modification is because we are not interested in the likelihood of m and conditioning on such variables simplifies the resulting model.
Namely,
[Equation]
where p ( y , u | z , θ ) : = p ( m ) · Gauss d y ( y | v ) · (cid:89) j ∈
[ d x ]: m j =
Gauss 1 (˜ x j | v ) , q ( z | y , u , ϕ ) := Gauss-NN 10 , 256 ( z | y ⊕ ˜ x ⊕ m ) , v :=
( ReLU ◦ Linear 256 )
( z ⊕ m ) and z ∼ q ( z | y , ˜ x , m , ϕ ) .
For VAE*, replace v with v ∗ :
= ( ReLU ◦ Linear 256 )( z ) .
Here, p ( m ) is an arbitrary density function of m , which is constant with respect to the parameters ( θ, ϕ ) and ignored in the gradient computation.
[Table]
{+2.2 except the missing-value ratio is p = 
+}
.instead of p = 0 .
CVAE.
The objective of CVAE is given according to Sohn et al.
[Equation]
where
[Equation]
and z ∼ q ( z | y , u , ϕ ) .[-[Table] 2.2 except the missing-value ratio is p = 
-]
DVAE, DVAE*.
The objective of DVAE is given by Equation 13, where [Equation]
Gauss [Equation] f [Equation]
ln p
[Equation] m )
+ Linear
[Equation]
g [Equation]
˜
[Equation] ω )) , g k ( t (1) , t (2) , t (3) , t (4) ; ω ) :
= ( ReLU ◦ Linear k )( t (1) ⊕ t (2) ⊕ t (3) ⊕ t (4) ) for t (1) , t (2) ∈ R d y and t (3) , t (4) ∈
R d x , ω denotes the shared parameter of ϕ, ψ, ξ , and v :=
( ReLU ◦ Linear 256 )
( z ⊕ m ) .
For DVAE*, replace v with v ∗ :
= ( ReLU ◦ Linear 256 )( z ) .
Here, p ( m ) is an arbitrary density function of m , which cancels out in the final objective of CELBO or CELBO-SP.
E A DDITIONAL R ESULTS
We show in Table 4 the results of the same experiment as in Section 4.2.2 except with the increased missing-value ratio p = 0 .
F A DDITIONAL D ISCUSSION ON R ELATED W ORK : M ISSING -D ATA H ANDLING
Arguably the most classic approach to the missing value problem is the two-step approach, also known as the imputation method (Enders, 2010).
This category includes traditional listwise or pairwise deletion methods, single imputation methods and multiple imputation methods.
The key feature of this approach is that one processes the incomplete features to get estimates of complete ones
ˆ x ≈ x in the first step and then performs prediction based on ˆ x .
The score-based methods (Jaakkola & Haussler, 1999; Holub et al., 2005; Perina et al., 2009; ´ Smieja et al., 2018) can be considered as a generalization of the two-step approach developed in the machine learning literature.
With a score-based method, one first learns the generative distribution p θ ( x ) , which is used to extract some information called score , s .
The score is then fed to predictive models p
θ ( y | s ) .
Several drawbacks stem from the two-step nature of these methods.
In case of the imputation method, the predictor loses the information whether each element of ˆ x is original or imputed.
This makes it difficult to estimate the uncertainty resulted from the feature corruption.
Moreover, even though the uncertainty [Table] problem can be addressed with the score-based method, learning good score representations requires solving optimization problems possibly irrelevant to the original prediction problem and thus it may compromise the performance.
Another category of missing feature handling fully utilizes generative models.
One of such method is referred to as the full information maximum likelihood methods (see also Secion 4, Enders (2010)), where the joint complete-data distribution p θ ( y , x ) is explicitly modeled and marginalized over the corrupted elements of features to obtain the objective ln p θ ( y , u ) .
This approach may suffer from unnecessarily performance degradation as in the score-based approach, since the objective contains the generative term of u ,
ln p θ ( y , u ) = ln p θ ( y | u ) + ln p θ ( u ) .
The third approach is the discriminative approach.
Specifically, tree-based models such as gradient boosting trees are able to naturally handle incomplete features (Twala et al., 2008).
Moreover, it is also recommended in (Kuhn et al., 2013) to encode missingness as a distinct feature, i.e., treat the concatenation of the corrupted feature and the mask vectors,
˜ x ⊕ m , as the input to the predictive models.
The advantage of this approach is that the resulting objective function is coherent with the goal of predictive risk minimization, i.e., there is no generative term unlike in the two-step and generative approach.
However, there is no trivial way to inform the model that m actually indicates missing features.
Therefore it may take extra samples to learn the meaning of m by itself.
Finally, the DIG approach can be thought of as a hybrid of the generative and discriminative approaches.
With DIG, the data is modeled with joint distribution p θ ( y , u ) , but the learning objective is the conditional evidence ln p θ ( y | u ) , which is computed from Bayes’ rule.
Therefore, it naturally incorporates the information of missingness and is directly trained to maximize the predictive performance.
The first application of the DIG strategy in the context of incomplete feature is Ghahramani & Jordan (1994), which was followed by Smola et al.
with a kernel-based generalization.
It is crucial in their results that the complete-data model p θ ( y , x ) is a exponential family so that the objective function is optimized with the EM algorithm (Dempster et al., 1977).
A tractable approximation algorithm for the case of Gaussian processes is derived by Pacheco et al.
(2014) with the combination of the variational inference (Jordan et al., 1999) and the expectation propagation (Minka, 2001) framework.
As opposed to their method, our focus is on black-box variational inference algorithm applicable to a variety of models.
See Table 5 for the summary of the comparison.
G M ODELING THE MAR P ROCESS
In this section, we give a method of modeling the third type of the feature-corruption processes, the
MAR process.
Let O ⊂ [ d x ] be an index set on which the feature is always observed (i.e., m j =for all j ∈ O ) and define ˜ x O :=
{ ˜ x j : j ∈ O} and ˜ x ¬O :=
{ ˜ x j : j ∈ [ d x ] \ O} .
Then, the MAR process is modeled with
[Equation]
where p ( ˜ x O , m ) will cancel out as well as p
[Table]
H A DDITIONAL J USTIFICATION AND L IMITATION OF THE S URROGATE T RANSFORM
We present another justification of the CELBO-SP maximization (i.e., optimization of Equation 15) as an alternative to the CELBO maximization (i.e., optimization of Equation 13).
Recall that
CELBO-SP is derived by the surrogate transform T SP and Proposition 2 shows a desirable property of T SP viewing it as an operator acting on the parameters .
In this section, we show another desirable property of T SP viewing it as an operator acting on the objective function .
More specifically, we view T SP as a mapping from ˆ L CELBO ( y | u ) to ˆ L CELBO-SP ( y | u )
=
ˆ L CELBO ( y | u ) ◦ T SP and discuss the relationship of these objective functions.
Let ζ := ( ϕ, ψ, ξ ) denote the tuple of the variational parameters for brevity and Ω be the set of the parameters ( θ, ζ ) on which we perform the optimization.
Let θ ∗ denote the true generative parameter of ( y , u ) , which is not necessarily contained in Ω .
Finally, define the discrepancy function of ( θ, ζ ) ∈ Ω with respect to θ ∗ by
[Equation]
where D KL ( y | u ) ( θ ∗ ∥ θ ) :
= E ( y , u ) ∼ θ ∗ [ln p ( y | u ,θ ∗ ) p
( y | u ,θ ) ] is the conditional Kullback–Leibler divergence of θ ∗ and θ and ∆( θ, ζ ) =
∆( θ, ϕ, ψ, ξ ) is the gap function defined just after Equation 12.
Note that it is nonnegative and takes zero if and only if both θ and ζ are correct with respect to θ ∗ , i.e., δ ( θ ∗ ∥ θ, ζ )
{+= 
+}
if and only if
[Equation]
for all y , u and z .
Note that the correctness of ( θ, ζ ) is measured with respect to the predictive form of the true model p ( y | u , θ ∗ ) , not the generative form p ( y , u | θ ∗ ) , which is an essence of DIG.
In other words, δ ( θ ∗ ∥ θ, ζ ) measures the predictive discrepancy of ( θ, ζ ) from θ ∗ .
Justification of CELBO maximization.
Observe that
[Equation]
where h ( y | u ) :
= E θ ∗ [ − ln p ( y | u , θ ∗ )] is the differential entropy of y given u .
Since h ( y | u ) is independent of ( θ, ζ ) , the CELBO maximization is justified as the discrepancy-function minimization,
[Equation]
Justification of CELBO-SP maximization as surrogate.
Similarly, we have
[Equation]
where δ SP ( θ ∗ ∥ θ, ζ ) :
= δ ( θ ∗ ∥ T SP ( θ, ζ )) is referred to as the transformed discrepancy function.
Then, the CELBO-SP maximization is seen as the transformed-discrepancy-function minimization,
[Equation]
⇔
[Equation]
Moreover, the transformed discrepancy function is consistent with the original discrepancy function in the following sense.
Proposition 5.
For all θ ∗ , θ and ζ ,
[Equation]
Proof.
It is shown as a corollary of Proposition 2.
In other words, δ SP ( θ ∗ ∥ θ, ζ ) also measures the predictive discrepancy of ( θ, ζ ) from θ ∗ in a different way.
This justifies the CELBO-SP maximization as a surrogate of the CELBO maximization.
Limitation of CELBO-SP maximization as surrogate.
A property stronger than the consistency (Equation 18) is the domination of the discrepancy function.
Here, we say δ SP dominates δ if there exists C < ∞ such that
[Equation]
for all θ ∗ and ( θ, ζ ) ∈ Ω .
If the domination holds, then the CELBO-SP maximization implies (in expectation) the CELBO maximization up to a multiplicative constant.
Unfortunately, however, this is not the case in general.
It is partly by design since the transformed discrepancy function δ SP ( θ ∗ ∥ θ, ζ ) is derived as a result of suppressing the divergence of the density ratio w θ,ψ,ξ ( u , z ) , which also causes the divergence of the original discrepancy function δ ( θ ∗ ∥ θ, ζ ) .
The following proposition shows that there exists no such constant C < ∞ satisfying Equation 19.
Proposition 6.
There exists a triple ( θ ∗ , θ, ζ ) such that δ ( θ ∗ ∥ θ, ζ ) = ∞ and δ SP ( θ ∗ ∥ θ, ζ ) < ∞ .
Proof.
It suffices to take ( θ ∗ , θ, ζ ) such that w αθ,ψ,ξ ( u , z ) is not integrable with the density q ( z | u , ψ ) , but { w θ,ψ,ξ ( u , z ) G ( w θ,ψ,ξ ( u , z )) } α is integrable with the same density.
For example, assume z takes a value in a Euclid space and take p ( z | u , θ ) ∝ exp( − ∥
z ∥ ) and q ( z | u , ψ ) ∝ exp( − ∥
z ∥ 2 ) .