EDDI: E FFICIENT D YNAMIC D ISCOVERY OF H IGH -V ALUE I NFORMATION WITH P ARTIAL VAE
Anonymous authors
Paper under double-blind review
A BSTRACT
Making decisions requires information relevant to the task at hand.
Many real-life decision making situations allow acquiring further relevant information at a speciﬁc cost.
For example, in assessing the health status of a patient we may decide to take additional measurements such as diagnostic tests or imaging scans before making a ﬁnal assessment.
More information that is relevant allows for better decisions but it may be costly to acquire all of this information.
How can we trade off the desire to make good decisions with the option to acquire further information at a cost?
To this end, we propose a principled framework, named EDDI (Efﬁcient
Dynamic Discovery of high-value Information), based on the theory of Bayesian experimental design.
In EDDI we propose a novel partial variational autoencoder (Partial VAE), to efﬁciently handle missing data over varying subsets of known information.
EDDI combines this Partial VAE with an acquisition function that maximizes expected information gain on a set of target variables.
EDDI is efﬁcient and demonstrates that dynamic discovery of high-value information is possible; we show cost reduction at the same decision quality and improved decision quality at the same cost in benchmarks and in two health-care applications.
We believe there is great potential for realizing these gains in real-world decision support systems.
I NTRODUCTION
Imagine that a person walks into a hospital with a broken arm.
The ﬁrst question from health-care personnel would be: “How did you break the arm?”
instead of “Do you have a cold?”
, because the answer reveals relevant information for this patient.
Human experts dynamically acquire information based on the current understanding of the situation.
Automating this human expertise of asking relevant questions is difﬁcult.
In other applications such as online questionnaires for example, most existing online questionnaire system either present exhaustive questions (Lewenberg et al., 2017;
Shim et al., 2018) or use extremely time-consuming human labeling work to manually build a decision tree for a reduced number of questions (Zakim et al., 2008).
This wastes the valuable time of experts or users (patients).
An automated solution for personalized dynamic acquisition of information has great potential to save much of this time in many real-life applications.
What are the technical challenges to build an intelligent information acquisition system?
Missing data is a key issue : taking the questionnaire scenario as an example, at any point in time we only observe a small subset of answers yet have to reason about possible answers for the remaining questions.
We thus need an accurate probabilistic model that can perform inference given a variable subset of observed answers.
Another key problem is deciding what to ask next : this requires assessing the worth of each possible question or measurement, the exact computation of which is intractable.
However, compared to current active learning methods we select individual features, not instances; therefore, existing methods are not applicable.
In addition, these traditional methods are often not scalable to the large volume of data available in many practical cases (Settles, 2012).
We propose the EDDI (Efﬁcient Dynamic Discovery of high-value Information) framework as a scalable information acquisition system for any given task.
We assume that information acquisition is always associated with cost.
Given a task, such as estimating the costumers’ experience or assessing population health status, we dynamically decide which piece of information to acquire next.
The framework is very general, and the information can be presented in any form such as answers to questions, or values of a lab test.
Our contributions are:
• We propose a novel efﬁcient information acquisition framework, EDDI (Section 3).
To enable EDDI, we contribute technically: 1.
A partial amortized inference method with different speciﬁcations for the inference network (Section 3.2).
We extend a current amortized inference method, the variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014), to account for partial observations.
The resulting method, which we call Partial VAE, is inspired by the set formulation of the data (Qi et al., 2017; Zaheer et al., 2017).
Partial VAE, as a probabilistic framework, is highly scalable, and serves as the base for the EDDI framework.
However, Partial VAE itself is generic and can be used on its own as a non-linear probabilistic framework for missing-data imputation.
An information theoretic acquisition function with an efﬁcient approximation, yielding a novel variable-wise active learning method (Section 3.3).
Based on the partial VAE, we select the unobserved variable which contributes most to the task, such as health assessment, evaluated using the mutual information.
This acquisition function does not have an analytical solution and we derive an efﬁcient approximation.
• We demonstrate the performance of EDDI on various settings, and apply it in real-life health-care scenarios (Section 4).
We ﬁrst show the superior performance of the Partial VAE framework on an image inpainting task (Section 4.1).
We then use 6 different datasets from the Machine Learning repository of University of Irvine (UCI) (Dheeru & Karra Taniskidou, 2017) to demonstrate the behavior of EDDI, comparing with multiple baseline methods (Section 4.2).
Finally, we evaluate EDDI on two real-life health-care applications: risk assessment in intensive care (Section 4.3) and public health assessment with national survey (Section 4.4), where traditional methods without amortized inference do not scale.
EDDI shows clear improvements in these two applications.
R ELATED WORK
EDDI requires a method that handles partially observed data to enable dynamic variable wise active learning.
We thus review related methods for handling partial observation and doing active learning.
2. P ARTIAL O BSERVATION
Missing data entries are common in many real-life applications, which has created a long history of research on the topic of dealing with missing data (Rubin, 1976;
Dempster et al., 1977).
We describe existing methods below:
Traditional methods without amortization Prediction based methods have shown advantages for missing value imputation (Scheffer, 2002).
Efﬁcient matrix factorization based methods have been recently applied (Keshavan et al., 2010; Jain et al., 2010; Salakhutdinov & Mnih, 2008), where the observations are assumed to be able to decompose as multiplication of low dimensional matrices.
In particular, many probabilistic frameworks with various distribution assumptions (Salakhutdinov &
Mnih, 2008; Blei et al., 2003) have been used for missing value imputation (Yu et al., 2016; Hamesse et al., 2018) and also recommender systems where unlabeled items are predicted (Stern et al., 2009;
Wang & Blei, 2011; Gopalan et al., 2014).
The probabilistic matrix factorization method has been used in the active variable selection framework, the dimensionality reduction active learning model (DRAL),(Lewenberg et al., 2017).
These traditional methods suffer from limited model capacity since they are commonly linear.
Additionally, they do not scale to large volumes of data and thus are usually not applicable in real-world applications.
For example, Lewenberg et al.
(2017) test the performance of their method with a single user due to the heavy computational cost of traditional inference methods for probabilistic matrix factorization.
Utilizing Amortized Inference The amortized inference (Kingma & Welling, 2014; Rezende et al., 2014; Zhang et al., 2017) has signiﬁcantly improved the scalability for probabilistic models such as variational autoencoders (VAEs).
In the case of partially observed data, amortized inference is particularly of interest due to the need of speeding up test time applications.
Wu et al.
(2018) employ traditional non-amortized inference in order to perform partial inference of a pretrained VAE during test time.
Amortized inference is only used during training, assuming the training dataset is fully observed.
During test time, the traditional inference is used to infer missing data entries from the partially observed dataset using the pre-trained model.
In this way, only training time is reduced.
The model is restrictive since it is not scalable in the test time and the fully observed training set is not available for many applications.
Nazabal et al.
(2018) uses zero imputation (ZI) for amortized inference for both training and test sets with missing data entries.
ZI is a generic and straightforward method that ﬁrst ﬁlls the missing data with zeros, and then feeds the imputed data as input for the inference network.
The drawback of zero imputation is that it introduces bias when the data are not missing completely at random which leads to not well-learned model.
We also observe artifacts when using it for the image inpainting task.
In the end, independent of our work, Garnelo et al.
(2018) explore interpreting variational autoencoder (amortized inference) as stochastic processes, which also handles partial observation per se.
2. A CTIVE L EARNING
Traditional Active Learning Active learning, also referred to as experimental design, aims to obtain optimal performance with fewer selected data (or experiments) (Lindley, 1956; MacKay, 1992; Settles, 2012).
Traditional active learning aims to select the next data point to label.
Many information theoretical approaches have shown promising results in various settings with different acquisition functions (MacKay, 1992; McCallumzy & Nigamy, 1998;
Houlsby et al., 2011).
These methods commonly assume that there exist fully observed data, and the acquisition decision is instance wise.
Little work has dealt with missing values within instances.
Zheng & Padmanabhan (2002) deal with missing data values by imputing with traditional non-probabilistic methods (Little & Rubin, 1987)
ﬁrst.
It is still an instance-wise active learning framework.
Different from traditional active learning, our proposed framework aims to for perform variable-wise active learning for each instance.
In this setting, information theoretical acquisition functions need a new design as well as non-trivial approximations.
The most closely related work is the aforementioned
DRAL (Lewenberg et al., 2017), which deals with variable-wise active learning for each instance.
Active Feartue Acquisition (AFA) Active sequential feature selection is of great need, especially in cost-sensitive applications.
Thus, many methods have also been applied and resulted in the class of methodologies called Active Feature Acquisition (AFA) (Melville et al., 2004; Saar-Tsechansky et al., 2009; Thahir et al., 2012; Huang et al., 2018).
For instance, Melville et al.
; Saar-Tsechansky et al.
(2009) have designed objectives to select any feature from any instance to minimize the cost to archive high accuracy.
The proposed framework is very general.
However, the problem setting of AFA methods are entirely different from our active variable selection problem: AFA mainly studies the optimization of optimal training set that would result in the best classiﬁer (model), under limited budget of costs, while our framework studies a slightly different problem: given a pretrained model, how to identify and acquire high value information with minimal costs.
Hence, AFA can not be directly applied.
Also, AFA requires fully observed variables at test time, while our framework does not require this assumption. Last but not the least, the realization of these framework relies on various heuristics and suffer from limited scalability.
M ETHOD
In this section, we ﬁrst formalize the active variable selection problem that we aim to solve.
Then, we present our Partial VAE to model and perform inference on partial observations.
Finally, we complete the EDDI framework by presenting our acquisition function and estimation method.
3. P ROBLEM FORMULATION
The core problem that we address in this paper is the following active variable selection problem.
Let x =
[ x 1 ,..., x | I | ] be a set of random variables with probability density p ( x ) .
Furthermore, let a subset of the variables x
O , O ⊂ I , be observed while the variables x U , U =
I \
O , are unobserved.
We assume that we can query the value of variables x i for i ∈ U .
The goal of active variable selection is to query a sequence of variables in U with the goal of predicting a quantity of interest f ( x ) , as accurately as possible while simultaneously performing as little queries as possible, where f ( · ) can be any (random) function.
This problem, in the simpliﬁed myopic setting, can be formalized as that
( | O | , M + 1 ) ee 2 ... e i e | O | xx 2 ...
x i x | O | h h shared h h g ( 1 , K ) c E N C
OD E R (a) Partial VAE PN setting ( | O | , M ) e... e | O | xx | O | × × h shared h g ( 1 , K ) c E N C OD E R (b) Parial VAE PNP setting of proposing the next variable x i ∗ to be queried by maximizing a reward function R , i.e.
[Equation] where R ( i | x O ) quantiﬁes the merit of our prediction of f ( · ) given x 0 and x i . Furthermore, the reward can quantify other properties important to the problem, e.g. the cost of acquiring x i .
3. P ARTIAL A MORTIZATION OF I NFERENCE Q UERIES
We ﬁrst introduce how to establish a generative probabilistic model of random variables x , that is capable of handling unobserved (missing) variables x U with variable size.
Our approach to this, named the Partial VAE, is based on the Variational autoencoder (VAE), which enables inference to scale to large volumes of data.
VAE and amortized inference VAE deﬁnes a generative model where the data x are generated from latent variables z , deﬁned as p ( x , z ; θ ) = ∏ n p ( x n | z n ; θ ) p ( z n ) .
The data generation, p θ ( x n | z n ) , is realized by a deep neural network. To approximate the the posterior of the latent variable p θ ( z n | x n ) , VAE uses amortized variational inference. Speciﬁcally, it uses an encoder, which is another neural network with the data x n as input to produce the variational approximation of the posterior q ( z n | x n ; φ ) .
As traditional variational inference, VAE is trained by maximizing an evidence lower bound (ELBO), which is equivalent to minimize the KL divergence between p θ ( z n | x n ) and q ( z n | x n ; φ ) .
VAE is not directly applicable to data with missing values.
Consider a partitioning that divides the variables into observed variables x O and unobserved variables x U .
In this setting, we would like to efﬁciently and accurately infer p ( z | x O ) and p ( x U | x O ) .
One challenge in the above setting is that there are many possible partitioning of { U , O } , where the size of observed ratings might vary.
Therefore, classic approaches to train a VAE with variational bound and amortize inference networks are no longer directly applicable.
We propose to extend amortization to our partial inference situation.
Partial VAE
In a VAE, a factorized structure for p ( x | z ) is always assumed, i.e.
[Equation]
This implies that given z , the observed variables x O are conditionally independent of x U .
Therefore,
[Equation]
and inferences about x U can be reduced to inference about z .
Therefore, the key object of interest in this setting is p ( z | x O ) , i.e., the posterior over the shared latent variables
z given the observed variables x O .
Once knowledge about z is obtained, we can draw correct inferences about x U .
To approximate p ( z | x O ) we introduce an auxiliary variational inference network q ( z | x O ) and deﬁne a partial variational upper bound,
[Equation]
This bound, L partial , depends only on the observation x O , which could vary between different data points.
We call the auxiliary distribution q ( z | x O ) the partial inference net since it takes a set of partially observed variables x O whose length may vary.
Specifying q ( z | x O ) requires distribution over random partitioning { O , U } .
Amortized Inference with partial observations Inspired by the Point Net (PN) approach for point cloud classiﬁcation (Qi et al., 2017; Zaheer et al., 2017), we specify the approximate distribution q
( z | x O ) by a permutation invariant set function encoding , given by:
[Equation]
where | O | is the number of the observed variables, s d carries the information of the input identify e d and the input value x d .
There are many ways to deﬁne e d .
Naively, it could be the coordinates for points in the point cloud, and one-hot embedding of the number of questions in a questionnaire.
With different problem settings, it can be beneﬁcial to learn e as an embedding of the identity of the variable, either with or without an naive encoding as input.
In this work, we treat e as an unknown embedding, which is optimized during training process.
There are also different ways to construct s d . Concatenation, s d = [ e d , x d ] , is commonly used in computer vision applications (Qi et al., 2017). Such architecture is illustrated in Figure 1(a).
However, we note that the construction of s d can be ﬂexible. We propose to construct s = e d ∗ x d using elementwise multiplication, shown in Figure 1(b).
We show that this formulation generalizes naive Zero Imputation (ZI) VAE (Nazabal et al., 2018).
We call this approach Pointnet Plus (PNP) speciﬁcation of Partial VAE. The theoretical consideration of relating ZI to PNP is presented in Appendix C.1.
We then use a neural network h ( · ) to map input s from R M + 1 to R K , where M is the dimension of each e d , x d is a scalar, and K is the latent space size.
Key to the PN structure is the permutation invariant aggregation operation g ( · ) , such as max-pooling or summation.
In this way, the mapping c ( x O ) is invariant to permutations of elements of x O and x O can have arbitrary length.
Finally, the ﬁxed-size code c ( x O ) is fed into an ordinary amortized inference net, that transforms the code into the statistics of a multivariate Gaussian distribution to approximate p ( z | x O ) .
The procedure is illustrated in the ﬁrst dashed box in Figure 1, which is our basic Partial VAE method.
3. E FFICIENT D YNAMIC D ISCOVERY OF H IGH - VALUE I NFORMATION
We now cast the active variable selection problem (1) as adaptive Bayesian experimental design, utilizing p ( x U | x O ) inferred by Partial VAE.
Algorithm 1 summarize the EDDI framework.
Information Reward We designed a variable selection acquisition function in an information theoretical way following Bayesian experimental design (Lindley, 1956; Bernardo, 1979).
Lindley (1956) provides a generic formulation of Bayesian experimental design by maximizing the expected
Shannon information.
Bernardo (1979) generalizes it by considering the decision task context.
For a given task, we may be interested in statistics of some variables x φ , where x φ ⊂ x U .
Given a new instance (user), assume we have observed x O so far for this instance, and we need to select the next variable x
i (an element of x U \ φ ) to observe.
Following Bernardo (1979), We select x i by maximizing:
[Equation]
In our paper, we mainly consider the case that a subset of interesting observations represents the statistics of interest x φ .
Sampling x
i ∼ p ( x i | x o ) is approximated by x i ∼ ˆ p ( x i | x o ) , where ˆ p ( x i | x o ) is deﬁned by the following process in Partial VAE.
It is implemented by ﬁrst sampling z ∼ q ( z | x o ) , and then x i ∼ p ( x i | z ) .
The same applies for p ( x i , x φ | z ) appeared in Equation 9.
Efﬁcient approximation of the Information reward The Partial VAE allows us to sample x i ∼ p ( x i | x o ) .
However, the KL term in Equation 6,
[Equation]
is intractable to evaluate since both p ( x φ | x i , x o ) and p ( x φ | x o ) are intractable.
For high dimensional x φ , entropy estimation could be difﬁcult.
The entropy term (cid:82) x φ p ( x φ | x i , x o ) log p ( x φ | x i , x o ) depends on i hence cannot be ignored.
In the following, we show how to approximate this expression.
Our proposal is based on the observation that analytic solutions of KL-divergences are available under speciﬁc variational distribution families of q ( z | x O ) (such as the Gaussian distribution commonly used in VAEs).
Instead of calculating information reward in x space, we have shown that one can equivalently perform calculations in z space (cf.
Appendix A.1):
[Equation]
O repeat for
[Table]
Note that Equation 8 is exact.
Additionally, we use partial VAE approximation p ( z | x φ , x i , x o )
≈
q
( z | x φ , x i , x o ) , p ( z | x o )
≈ q
( z i | x o ) and p ( z | x i , x o )
≈ q
( z i | x i , x o ) .
This leads to the ﬁnal approximation of the information reward:
[Equation]
[Equation] (cid:3) . With this approximation, the divergence between q ( z | x i , x o ) and q ( z | x o ) can often computed analytically in Partial VAE setting, for example, under Gaussian parameterization.
The only Monte Carlo sampling required is the one set of samples x φ , x i ∼ p ( x φ , x i | x o ) that can be shared across different
KL terms in Equation 9.
E XPERIMENTS
We evaluate our proposed EDDI framework with various settings.
We ﬁrst assess the Partial VAE component of EDDI alone on an image inpainting task both qualitatively and quantitatively (Section 4.1).
We compare our proposed two PN-based Partial VAE with the zero-imputing (ZI) VAE (Nazabal et al., 2018).
Additionally, we modify ZI VAE to use s mask matrix indicating which variables are currently observed as input.
We name this method ZI-m VAE.
We then demonstrate the performance of the entire EDDI framework on datasets from the UCI repository (Section 4.2 ), as well as in two real-life application scenarios: Risk assessment in intensive care (Section 4.3) and public health assessment with national health survey (Section 4.4).
We compare the performance of EDDI, using four different Partial VAE settings, with three baselines.
The ﬁrst baseline is the random active feature selection strategy (denoted as RAND) which randomly picks the next variable to observe.
The second baseline method is the single best strategy (denoted as SING) which ﬁnds a single global optimal order of picking up variables.
This order is then applied to all data points.
SING uses the objective function as in Equation (9) to ﬁnd the optimal ordering by averaging over all the data.
4. I MAGE INPAINTING WITH P ARTIAL VAE
We evaluate the performance of Partial VAE with the image inpainting task, which is to ﬁll in the removed pixels in an image.
We perform the evaluation in two different settings: We remove the pixels at random in the ﬁrst setting, and remove a region of the pixels in the second setting.
Inpainting Random Missing Pixels We use MNIST dataset (LeCun, 1998) and remove pixels randomly for this task.
The same setting are used for all methods (see Appendix B.1 for details).
During training, we remove a random portion (uniformly sampled between 0% and 70%) of pixels.
We then impute missing pixels on a partially observed test set (constructed by removing 70% of the pixels).
The performance of pixel imputation is evaluated by test ELBOs on missing pixels.
The ﬁrst two rows in Table 1 show training and test ELBOs for all algorithms using this partially observed
[Figure] [Figure]
n
[Figure] t [Figure] li
[Figure] dataset. Additionally, we show ordinary VAE (VAE-full) trained on the fully observed dataset as an ideal reference.
Among all Partial VAE methods, the PNP approach performs best.
Inpainting Regions We then consider inpainting large contiguous regions of images.
It aims to evaluate the capability of the Partial VAEs to produce all possible outcomes with better uncertainty estimates.
With the same trained model as before, we remove the region of the upper 60% pixels of the image in the test set.
We then evaluate the average likelihoods of the models.
The last row of Table 1 shows the results of the test ELBO in this case.
PNP based Partial VAE performs better than other settings.
Note that given only the lower half of a digit, the number cannot be identiﬁed uniquely.
ZI (Figure 2(b)) fails to cover the different possible modes due to its limitation in posterior inference.
ZIm (Figure 2(c)) is capable of producing multiple modes.
However, some of the generated samples are not consistent with the given part (i.e., some digits of 2 are generated).
Our proposed PN (Figure 2(d)) and PNP Figure 2(e)) are capable of recovering different modes, and are consistent with observations.
4. EDDI ON UCI DATASETS
Given the effectiveness of our proposed Partial VAE, we now demonstrate the performance of our proposed EDDI framework in comparison with random selection (RAND) and single optimal ordering (SING).
We ﬁrst apply EDDI on 6 different UCI datasets (cf.
Appendix B.2) (Dheeru &
Karra Taniskidou, 2017).
We report the results of EDDI with all these four different speciﬁcations of
Partial VAE (ZI, ZI-m, PN, PNP).
All Partial VAE are ﬁrst trained on partially observed UCI datasets where a random portion of variables is removed.
We actively select variable for each test point starting with empty observation x o = /0.
In all UCI datasets, We randomly sample 10% of the data as the test set.
All experiments repeated for ten times.
Taking PNP based setting as an example, Figure 3 shows the negative test log likelihood on x φ for each variable selection step with three different datasets, where x φ is deﬁned by the UCI task.
We call this curve the information curve (IC) .
We see that EDDI can obtain information efﬁciently.
It archives the same negative test log likelihood with less than half of the variables.
Single optimal ordering also improves upon random ordering.
However, it is less efﬁcient compared with EDDI since EDDI perform active learning for each data instance which is “personalized”.
Figure 4 shows [Table] [Table] an example of the decision processes using EDDI and SING.
The ﬁrst step of EDDI overlaps largely with SING.
From the second step EDDI makes “personalized” decisions
We also present the average performance among all datasets with different settings.
The area under the information curve (AUIC), ∑ t − log p ( x φ | x O t ) , can then be used to compare the performance across models and strategies.
Smaller AUIC value (could be positive or negative) indicates better performance.
However, due to different datasets have different scales of test likelihoods and different numbers of variables (indicated by steps), it is not fair to average the AUIC across datasets to compare overall performances.
We thus deﬁne average ranking of AUIC that compares 12 methods (indexed by i ) averaging these datasets as: r i =
∑ j N j ∑ 6 j = 1 ∑ N j k = 1 r ijk , i = 1 ,.., 16 .
These 12 methods are cross combinations of four Partial VAE models with three variable selection strategies.
r i is the ﬁnal ranking of i th combination, r ijk is the ranking of the i th combination (based on AUIC value) regarding the k th test data point in the j th UCI dataset, and N j the size of the j th UCI dataset. This gives us 6 ∑ j N j different rankings.
Finally, we simply compute the mean and standard error statistics based on these rankings.
Table 2 summarize the average ranking results.
We provide additional statistical signiﬁcance test (Wilcoxcon signed-rank test for paired data) in Appendix B.2.3.
We can conclude that EDDI outperforms other variable selection order in all different Partial VAE settings.
Among different partial VAE settings, PNP/PN-based settings perform better than ZI-based settings.
Comparison with non-amortized method Additionally, we compare EDDI to DRAL (Lewenberg et al., 2017) which is the state-of-the-art method for the same problem setting.
As discussed in
Section 2, DRAL is linear and requires high computational cost.
The DRAL paper only tested their method on a single test data point due to its limitation on computational efﬁciency.
We compare
DRAL with EDDI on Boston Housing dataset with ten randomly selected test points here.
Results are shown in Figure 5, where EDDI signiﬁcantly outperforms DARL thanks to more ﬂexible Partial VAE model.
Additionally, EDDI is 1000 times more efﬁcient than DARL as shown in Table 3.
4. R ISK ASSESSMENT WITH MIMIC-III
We now apply EDDI to risk assessment tasks using the Medical Information Mart for Intensive Care (MIMIC III) database (Johnson et al.
MIMIC III is the most extensive publicly available clinical database, containing real-world records from over 40,000 critical care patients with 60,
ICU stays.
The risk assessment task is to predict the ﬁnal mortality.
We preprocess the data for this task following Harutyunyan et al.
This results in a dataset of 21139 patients.
We treat the ﬁnal mortality of a patient as a Bernoulli variable.
For our task, we focus on variable selection, which corresponds to medical instrument selection.
We thus further process the time series variables into static variables based on temporal averaging.
[Table]
[Table]
Figure 6 shows the information curve of different strategies
[Table]
using PNP based Partial VAE as an example (more results in Appendix B.3).
Table 4 shows the average ranking of AUIC with different settings.
In this application, EDDI signiﬁcantly outperforms other variable selection strategies in all different settings of Partial VAE
[Table]
and PNP based setting performs best.
4. P UBLIC H EALTH A SSESSMENT WITH NHANES
Finally, we apply our methods to public health assessment using NHANES 2015-2016 data cdc (2005).
NHANES is a program with adaptable components of measurements, to assess the health and nutritional status of adults and children in the United States.
Every year, approximately thousands individuals of all ages are interviewed in their homes and complete the health examination component of the survey.
This 2015-2016 NHANES data contains three major sections, the questionnaire interview, examinations and lab tests for 9971 subjects in the publicly available version of this cycle.
In our setting, we consider the whole set of lab test results (139 dimensions of variables) as the target variable of interest x φ since they are expensive and reﬂects the subject’s health status, and we active select the questions from the extensive questionnaire (665 variables).
In NHANES, the entire questionnaire is divided into 73 different groups.
In practice, questions in the same group are often examined together.
Therefore, we perform active variable selection on the group level: at each step, the algorithm will be selecting one group to observe.
This is more challenging than the experiments in previous sections since it requires the generative model to simulate a group of unobserved data in Equation (9) at the same time.
When evaluating test likelihood on the target variable of interest, we treat variables in each group equally.
For a fair comparison, the calculation of the area under the information curve (AUIC) is weighted by the size of the group chosen by the algorithms.
Speciﬁcally, AUIC is calculated after spline interpolation.
The information curve plots in
Figure 7, together with Table 5 of AUIC statistics show that our EDDI outperforms other baselines.
This experiment shows that EDDI is capable of performing active selection on a large pool of grouped variables to estimate a high dimensional target.
C ONCLUSION
In this paper, we present EDDI, a novel and efﬁcient framework for dynamic active variable selection for each instance.
Within the EDDI framework, we propose Partial VAE which performs amortized inference to handle missing data.
Partial VAE alone can be used as a non-linear computational efﬁcient probabilistic imputation method.
Based on Partial VAE, we design a variable wise acquisition function for EDDI and derived corresponding approximation method.
EDDI has demonstrated its effectiveness on active variable selection tasks across multiple real-world applications.
In the future, we would extend the EDDI framework to handle more complicated scenarios, such as time-series, or the cold-start situation.
A A DDITIONAL D ERIVATIONS A. I NFORMATION REWARD APPROXIMATION
In our paper [Equation] given the VAE model p ( x | z ) and a partial inference network q ( z | x o ) , the experimental design problem is formulated as maximization of the information reward:
[Equation]
Where p ( x φ | x i , x o ) = [Equation] z p ( x φ | z ) q ( z | x i , x o ) , p ( x φ | x o ) = (cid:82) z p ( x φ | z ) q ( z | x o ) and q ( z | x o ) are approximate condition distributions given by partial VAE models.
Now we consider the problem of directly approximating R ( i , x o ) .
Applying the chain rule of KL-divergence [Equation] we have:
[Equation]
Using again the KL-divergence chain rule on D KL ( p ( x φ , z | x i , x o ) || p ( x φ , z | x o ))
[Equation]
we have:
[Equation]
KL [Equation] The KL-divergence term in the reward formula is now rewritten as follows,
[Equation]
One can then plug in the partial VAE inference approximation:
[Equation]
Finally
[Equation]
the information reward is now approximated as:
[Equation]
This new objective tries to maximize the shift of belief on latent variables z by introducing x i , while penalizing the information that cannot be absorbed by x φ (by the penalty term
D KL ( q ( z | x φ , x i , x o ) ||
q ( z | x φ , x o )) ).
Moreover, it is more computationally efﬁcient since one set of samples x φ , x i ∼ p ( x φ , x i | x o ) can be shared across different terms, and the KL-divergence between common parameterizations of encoder (such as Gaussians and normalizing ﬂows) can be computed exactly without the need for approximate integrals.
Note also that under approximation p ( z | x φ , x i , x o )
≈
q
( z | x φ , x i , x o ) , p ( z | x i , x o )
≈
q
( z | x i , x o ) , p ( z | x o )
≈
q
( z | x o ) , sampling x i ∼ p ( x i | x o ) is approximated by x i ∼ ˆ p ( x i | x o ) , where ˆ p ( x i | x o ) is deﬁned by the following process in Partial VAE.
It is implemented by ﬁrst sampling z ∼ q ( z | x o ) , and then x i ∼ p ( x i | z ) .
The same applies for p ( x i , x φ | z ) .
B A DDITIONAL E XPERIMENTAL R ESULTS B. I MAGE INPAINTING B.1. P REPROCESSING AND MODEL DETAILS
For our MNIST experiment, we randomly draw 10% of the whole data to be our test set.
Partial VAE models (ZI, ZI-m, PNP and PNs) share the same size of architecture with 20 dimensional diagonal
Gaussian latent variables: the generator (decoder) is a 20-200-500-500 fully connected neural network with ReLU activations (where D is the data dimension, D = 784).
The inference nets (encoder) share the same structure of D-500-500-200-40 that maps the observed data into distributional parameters of the latent space.
For the PN-based parameterizations, we use a 500 dimensional feature mapping h parameterized by a single layer neural network, and 20 dimensional ID vectors e i (see Section 3.2) for each variable.
We choose the symmetric operator g to be the basic summation operator.
During training, we apply Adam optimization (Kingma & Ba, 2015) with default hyperparameter setting, learning rate of 0.001 and a batch size of 100.
We generate partially observed MNIST dataset by adding artiﬁcially missingness at random in the training dataset during training.
We ﬁrst draw a missing rate parameter from a uniform distribution U ( 0 , 0 . 7 ) and randomly choose variables as unobserved.
This step is repeated at each iteration.
We train our models for 3K iterations.
B.1. I
[Figure] B. UCI DATASETS
We applied EDDI on 6 UCI datasets; Boston Housing, Concrete compressive strength, energy efﬁciency, wine quality, Kin8nm, and Yacht Hydrodynamics.
The variables of interest x φ are chosen to be the target variables of each UCI dataset in the experiment.
B.2. P REPROCESSING AND MODEL DETAILS
All data are normalized and then scaled between 0 and 1.
For each of the 10 - in total- repetitions, we randomly draw 10% of the data to be our test set.
Partial VAE models (ZI, ZI-m, PNP and PNs) share the same size of architecture with 10 dimensional diagonal Gaussian latent variables: the generator (decoder) is a 10-50-100-D neural network with ReLU activations (where D is the data dimensions).
The inference nets (encoder) share the same structure D-100-50-20 that maps the observed data into distributional parameters of the latent space.
For the PN-based parameterizations, we further use adimensional feature mapping h parameterized by a single layer neural network and 10 dimensional
ID vectors e i (please refer to section 3.2) for each variable.
We choose the symmetric operator g to be the basic summation operator.
As in the image inpainting experiment, we apply Adam optimization during training with default hyperparameter setting, and a batch size of 100 and ingest random missingness as before.
We trained our models for 3K iterations.
During active learning, we draw 50 samples in order to estimate the expectation under x φ , x i ∼ p ( x φ , x i | x o ) in Equation (8).
Negative likelihoods of the target variable is also estimated usingsamples of x φ ∼ p ( x φ | x o ) through p ( x φ | x o )
≈ 1 M
∑ Mm = 1 p ( x φ | z m ) , where z m ∼ q ( z | x o ) .
B.2. T ABLES ON A REA UNDER THE INFORMATION CURVE (AUIC)
In addition to the area under the information curve (AUIC) ranking metric provided in the paper, the average area under the information curve (Avg. AUIC) on each dataset can also be used to compare the performance across models and strategies.
AUIC is deﬁned to be ∑ t − log p ( x φ | x O t ) , where x O t is the basket of variables observed at step t .
By deﬁnition, smaller AUIC value (could be positive or negative) indicates better performance.
We present the AUIC for each dataset in Table 6, 7, 8, 9, 10, and 11 2 .
Readers might have found that it seems that the avg.
AUIC results in Tables 6 - 11 contradicts the avg.
ranking of AUIC results in Table 2 of the main text. However, this is not the case. In Tables Tables 6 - 11, AUIC numbers only provide a simpliﬁed statistics of marginal distributions of each method’s performance.
Here, the distribution of performance is deﬁned by ﬁrst sample a data point from the data distribution, and then we obtain the performance of a active learning method of interest by evaluating its performance (AUIC) on this single data point. On the contrary, the average AUIC ranking measure actually takes into account the joint distributions of the performance of all methods, since ranking is a function of the performance of all methods. With this additional information of correlations, this gives a more accurate evaluation regarding the actual performance of different methods. Notably, in practical scenario of active variable selection, the latter setting is obviously more sensible and fare.
The above conjecture is further validated and conﬁrmed by applying the nonparametric statistical test, namely the Wilcoxcon signed-rank signiﬁcance test on the performance of different methods, which are detailed in Appendix B.2.3. Wilcoxon test is a very powerful statistical test which includes the information of the joint distribution in paired samples. In our case, the term paired samples refers to the situation that different algorithms are evaluated on exactly the same set of test data points, which introduces correlations between the performances of different algorithms. [Table]
[Table]
[Table] [Table] [Table] [Table]
B.2. S TATISTICAL SIGNIFCANT TEST RESULTS
In this section
[Table]
we perform Wilcoxcon signed-rank signiﬁcance test on the performance of different methods, to support our result in Table 2.
Since Table 2 suggests that EDDI-PNP-Partial VAE is the best algorithm overall
[Table]
we set EDDI-PNP-Partial VAE as default and perform Wilcoxcon test between EDDI-PNP-Partial VAE and all other 15 different settings, to see whether the improvement is signiﬁcant.
Table 12 displays the corresponding p-value for each test.
It is obvious that in all 15 tests, the EDDI-PNP-Partial VAE results are signiﬁcant (compared with the standard α = 0 .cutoff).
This provides strong evidence that conﬁrms our results in Table 2 and our conjecture in
Appendix B.2.2.
[Table]
A DDITIONAL AUIC PLOTS OF PN, ZI AND ZI- M ON UCI DATASETS
Here we present additional plots of the information curve during active variable selection.
Figure 9 presents the results for the Boston Housing, the Energy and the Wine datasets and for the three approaches, i.e. PN, ZI and masked ZI.
[Figure]
This displays negative test log likelihood (y axis, the the better) during the course of active selection (x-axis). B.2. RMSE PLOTS OF PN, ZI AND ZI- M ON UCI DATASETS Here we present additional plots of the RMSE curves during active learning. Figure 10 presents the results for the Boston Housing, the Energy and the Wine datasets and for the three approaches, i.e
Boston Housing
[Figure]
B.2. C OMPARISONS BETWEEN EDDI AND LASSO- BASED METHOD
Here we present additional results of a new baseline, the LASSO-based feature selection.
This is not presented in the main text since LASSO is designed for a different problem setting. It requires fully observed data, and only works in regression problems with one dimensional outputs. Both MIMIC III and NHANES tasks do not fulﬁll these requirements.
Additionally, LASSO aims to select a global set of features to obtain the best performance instead of select the most informative feature given partially observed information, thus cannot be used in a sequential setting. We thus construct the LASSO feature selection baseline as follows for comparison: we ﬁrst apply LASSO regression on training dataset which is fully observed in these UCI datasets, and select the features (denoted by A ) that correspond to non-zero coefﬁcients. Then, during test time, LASSO strategy will observe the features one by one from A randomly.
When all variables selected by LASSO are already picked, we stop the feature selection progress.
Since LASSO does not support evaluation of model likelihood as well as it is linear, we use the corresponding partial-VAE (ZI,ZI-m,PNP,PN) to make predictions and evaluate the model log likelihood.
Figure 11 presents the results for the Boston Housing, the Energy and the Wine datasets as examples.
Full results of all UCI datasets are presented in Table 13.
Note that in Table 13, Wilcoxon signed-rank test is performed between EDDI and LASSO strategies for each Partial VAE models, respectively.
The results indicates that EDDI signiﬁcantly outperforms LASSO in all circumstances.
This is despite the fact that EDDI is a greedy sequential variable selection method that built upon partially observed data, while LASSO-baseline makes use of the information from fully observed data , and selects the set of variables in a non-greedy, global manner , which is often unrealistic in many pratical application settings.
[Figure]
[Figure]
[Table] [Figure] [Table] B.2. I LLUSTRATION OF DECISION PROCESS OF EDDI (B OSTON H OUSING AS EXAMPLE )
The decision process facilitated by the active selection of the variables (for the EDDI framework) is efﬁciently illustrated in Figure 12 and Figure 13 for the Boston Housing dataset and for the PNP and
PNP with single best ordering approaches, respectively.
For completeness, we provide details regarding the abbreviations of the variables used in the Boston dataset and appear both ﬁgures.
CR - per capita crime rate by town PRD - proportion of residential land zoned for lots over 25,000 sq.ft.
PNB - proportion of non-retail business acres per town.
CHR - Charles River dummy variable (1 if tract bounds river; 0 otherwise) NOC - nitric oxides concentration (parts per 10 million) ANR - average number of rooms per dwelling AOUB - proportion of owner-occupied units built prior toDTB - weighted distances to ﬁve Boston employment centres ARH - index of accessibility to radial highways TAX - full-value property-tax rate per $10, OTR - pupil-teacher ratio by town PB - proportion of blacks by town LSP - % lower status of the population
B. MIMIC-III
Here we provide additional results of our approach on the MIMIC-III dataset.
[Figure]
B.3. P REPROCESSING AND MODEL DETAILS
For our active learning experiments on MIMIC III datasets, we chose the variable of interest x φ to be the binary mortality indicator of the dataset.
All data (except the binary mortality indicator) [Figure] are normalized and then scaled between 0 and 1.
We transformed the categorical variables into real-valued using the dictionary deduced from (Johnson et al., 2016) that makes use of the actual medical implications of each possible values.
The binary mortality indicator are treated as Bernoulli variables and Bernoulli likelihood function is applied.
For each repetition (of the 5 in total), we randomly draw 10% of the whole data to be our test set.
Partial VAE models (ZI, ZI-m, PNP and
PNs) share the same size of architecture with 10 dimensional diagonal Gaussian latent variables: the generator (decoder) is a 10-50-100-D neural network with ReLU activations (where D is the data dimensions).
The inference nets (encoder) share the same structure of D-100-50-20 that maps the observed data into distributional parameters of the latent space.
Additionally, for PN-based parameterizations, we further use a 20 dimensional feature mapping h parameterized by a single layer neural network, and 10 dimensional ID vectors e i (please refer to section 3.2) for each variable.
We choose the symmetric operator g to be the basic summation operator.
Adam optimization and random missingness is applied as in the previous experiments.
We trained our models for 3K iterations.
During active learning, we draw 50 samples in order to estimate the expectation under x φ , x i ∼ p ( x φ , x i | x o ) in Equation (8).
Negative likelihoods of the target variable is also estimated using 50 samples of x φ ∼ p ( x φ | x o ) through p ( x φ | x o )
≈ 1 M
∑ Mm = 1 p ( x φ | z m ) , where z m ∼ q ( z | x o ) .
B.3. A DDITIONAL PLOTS OF ZI, PN AND ZI- M ON MIMIC III
Figure 14 shows the information curves of active variable selection on the risk assessment task for
MIMIC-III as produced by the three approaches, i.e. ZI, PN and masked ZI.
[Figure]
B. NHANES B.4. P REPROCESSING AND MODEL DETAILS
For our active learning experiments on NHANES datasets, we chose the variable of interest x φ to be the lab test result section of the dataset.
All data are normalized and scaled between 0 and 1.
For categorical variables, these are transformed into real-valued variables using the code that comes with the dataset, which makes use of the actual ordering of variables in questionnaire.
Then, for each repetition (of the 5 repetitions in total), we randomly draw 8000 data as training set and 100 data to be test set.
All partial VAE models (ZI, ZI-m, PNP and PNs) uses gaussian likelihoods, with an diagonal Gaussian inference model (encoder).
Partial VAE models share the same size of architecture with 20 dimensional diagonal Gaussian latent variables: the generator (decoder) is a 20-50-100-D neural network.
The inference nets (encoder) share the same structure of D-100-50-20 that maps the observed data into distributional parameters of the latent space.
Additionally, for PN-based parameterizations, we further use a 20 dimensional feature mapping h parameterized by a single layer neural network, and 100 dimensional ID vectors e i (please refer to section 3.2) for each variable.
We choose the symmetric operator g to be the basic summation operator.
Adam optimization and random missingness is applied as in the previous experiments.
We trained all models 1K iterations.
During active learning, 10 samples were drawn to estimate the expectation in
Equation (9).
Negative likelihoods of the target variable is also estimated using 10 samples. ( | O | , M + 1 ) ee... e i e | O | xx...
x i x | O | h ( 1 ) h ( 1 ) shared h ( 1 ) h ( 1 ) g ( 1 , K ) c ( 1 ) First recurrent step ( | O | , M + 1 + K ) c on ca t e n a t e ee... e i e | O | xx...
x
i x | O c ( 1 ) c ( 1 ) ...
c ( 1 ) c ( 1 ) h ( 2 ) h ( 2 ) shared h ( 2 ) h ( 2 ) g c ( 2 ) ( 1 , K )
Second recurrent step
The output c ( 2 ) is directly connected to the rest of the inference network in this case.
One use steps
To
B. PN/PNP MODEL STRUCTURE DETERMINATION : SHOULD WE USE RECURRENT EXTENSIONS
One straightforward extention of PN/PNP Partial VAEs proposed in this paper is, to generalize PN and PNP by recurrently reuse the code c to enlarge the capacity of PN:
Figure 15 shows the mechanism of the recurrent PN with two recurrent steps using concatenated s ( 1 ) d
= [ e d , x d ] as an example.
The ﬁrst step is the same as the PN setting with the K dimensional output c ( 1 ) , where ( 1 ) is the recurrent step index.
For the second step, we concatenate the learned c ( 1 ) to s ( 1 ) to form the new input for the next recurrent step s ( 2 ) d
= [ s ( 1 ) d , c ( 1 ) ] .
There can be arbitrary number of recurrent steps using the input s ( n ) d
= [ s ( n − 1 ) d , c ( n − 1 ) ] .
Within each recurrent step, the parameters for the neural network h ( n ) are shared, however, different steps have different parameters.
When n = 1, we recover the original PN partial VAE setting.
The question is, should we include the recurrent structure in our Partial VAE?
In this section, we present preliminary result for this purpose.
We perform plot runs of PN1, PN2, PN5, PNP1, PNP1,
PNP2, PNP5 (PN-x stands for PN model with x recurrent steps) on MNIST dataset for 300 iterations.
Other model settings are consistent with Section B.1.1.
Results of negative test log likelihoods are shown in Figure 16:
Based on Figure 16, the conclusion is: based on MNIST dataset along, by increasing the recurrent steps of PN, the performance roughly increases slightly.
Meanwhile, we can observe that increasing the recurrent steps does not improve PNP.
However, the difference is not signiﬁcant.
In section B.2.2, a comparison between recurrent PN (PN5) and vanilla PN
(PN1) is considered for each UCI dataset, see Table 6, 7, 8, 9, 10, and 11.
Additionally, the average ranking of AUIC of PN and PN1 is summarized in Table 14:
[Table]
It is clear that one can not conclude that on average, PN5 signiﬁcantly outperforms PN1.
Therefore, as far as active learning tasks under the settings in our experiments are considered, we believe the recurrent generalization of PNs will not be a crucial factor for boosting a performance.
C A DDITIONAL T HEORETICAL C ONTRIBUTIONS C. Z ERO IMPUTING AS A P OINT N ET
Here we present how the zero imputing (ZI) and PointNet (PN) approaches relate.
Zero imputation with inference net In ZI, the natural parameter of λ (e.g., Gaussian parameters in variational autoencoders) is approximated using the following neural network:
[Equation]
, where L is the number of hidden units, x is the input image with x i be the value of the i th pixel.
To deal with partially observed data x = x o ∪ x u , ZI simply sets all x u to zero, and use the full inference model f ( x ) to perform approximate inference.
PointNet parameterization The PN approach approximates the natural parameter λ by a permutation invariant set function
[Equation]
where s i =
[ x i , e i ] , e i is the I dimensional embedding/ID/location vector of the i th pixel, g ( · ) is a symmetric operation such as max-pooling and summation, and h ( · ) is a nonlinear feature mapping from R I
+ 1 to R K
(we will always refer h as feature maps ).
In the current version of the partial-VAE implementation, where Gaussian approximation is used, we set K = 2 H with H being the dimension of latent variables.
We set g to be the element-wise summation operator, i.e. a mapping from R KO to R K deﬁned by:
[Equation]
This parameterization corresponds to products of multiple Exp-Fam factors ∏ i ∈
O exp {−(cid:104)
h ( s i ) , Φ (cid:105)} .
From PN to ZI To derive the PN correspondence of the above ZI network we deﬁne the following
PN functions:
[Equation]
where h k ( · ) is the k th output feature of h ( · ) .
The above PN parameterization is also permutation invariant; setting L
=
I , θ l = w ( 1 ) l , ( w ( 0 ) l ) i =
( e i ) l the resulting PN model is equivalent to the ZI neural network.
Generalizing ZI from PN perspective In the ZI approach, the missing values are replaced with zeros.
However, this ad-hoc approach does not distinguish missing values from actual observed zero values.
In practice, being able to distinguish between these two is crucial for improving uncertainty estimation during partial inference.
One the other hand, we have found that PN-based partial VAE experiences difﬁculties in training.
To alleviate both issues, we proposed a generalization of the ZI approach that follows a PN perspective.
One of the advantages of PN is setting the feature maps of the unobserved variables to zero instead of the related weights.
As discussed before, these two approaches are equivalent to each other only if the factors are linear.
More generally, we can parameterize the PN by:
[Equation]
where NN 1 is a mapping from R I to R K deﬁned by a neural network, and NN 2 is a mapping from R K to R 2 H deﬁned by another neural network.
C. A PPROXIMATION D IFFICULTY OF THE A CQUISITION F UNCTION
Traditional variational approximation approaches provide wrong approximation direction when applied in this case (resulting in an upper bound of the objective R φ ( i , x O ) which we maximize).
Justiﬁcation issues aside, (black box) variational approximation requires sampling from approximate posterior q ( z | x O ) , which leads to extra uncertainties and computations.
For common proposals of approximation:
• Directly estimate entropy via sampling ⇒ problematic for high dimensional target variables • Using reversed information reward E x i ∼ p ( x i | x o )
[ D KL ( p ( x φ | x o ) || p ( x φ | x o , x i ))] , and then apply ELBO (KL-divergence)
⇒
This does not make sense mathematically, since this will result in upper bound approximation of the (reversed) information objective, this is in the wrong direction.
• Ranganath’s bound (Ranganath et al., 2016) on estimating entropy ⇒ gives upper bound of the objective, wrong direction.
• All the above methods also needs samples from latent space (therefore second level approximation needed).
C. C ONNECTION OF EDDI INFORMATION REWARD WITH BALD
We brieﬂy discuss connection of EDDI information reward with BALD (Houlsby et al., 2011) and.
MacKay’s work (MacKay, 1992).
Assuming the model is correct, i.e. q = p , we have
[Equation]
Note that based on McKay’s relationship between entropy and KL-divergence reduction
[Equation]
we have:
[Equation]
Similarly, we have
[Equation]
x o
[Equation]
, where MacKay’s result is applied to E x i ∼ p ( x i | x φ , x o ) (cid:2) D KL ( p ( z | x φ , x i , x o ) ||
p ( z | x φ , x o )) (cid:3) .
Putting everything together
[Equation]
we have
[Equation]
We can show that
[Equation]
which is exactly the conditional mutual information I (cid:2)
z , x φ | x i , x o (cid:3) used in BALD.
Therefore
[Equation]
our chain rule representation of reward function leads us to
[Equation]