K NOCKOFF GAN: G ENERATING K NOCKOFFS FOR F EATURE S ELECTION USING G ENERATIVE A DVERSARIAL N ETWORKS
A BSTRACT
Feature selection is a pervasive problem.
The discovery of relevant features can be as important for performing a particular task (such as to avoid overﬁtting in prediction) as it can be for understanding the underlying processes governing the true label (such as discovering relevant genetic factors for a disease).
Machine learning driven feature selection can enable discovery from large, high-dimensional, nonlinear observational datasets by creating a subset of features for experts to focus on.
In order to use expert time most efﬁciently, we need a principled methodology capable of controlling the False Discovery Rate.
In this work, we build on the promising Knockoff framework by developing a ﬂexible knockoff generation model.
We adapt the Generative Adversarial Networks framework to allow us to generate knockoffs with no assumptions on the feature distribution .
Our model consists of 4 networks, a generator, a discriminator, a stability network and a power network.
We demonstrate the capability of our model to perform feature selection, showing that it performs as well as the originally proposed knockoff generation model in the Gaussian setting and that it outperforms the original model in non-Gaussian settings, including on a real-world dataset.
I NTRODUCTION
Feature selection is a pervasive problem.
Often the goal is to discover features that are relevant to a particular outcome, either for the sake of discovery itself or to aid in prediction [16; 25].
When the focus is on discovery, feature selection methods typically focus on trying to control either the
Family-Wise Error Rate (FWER) or the False Discovery Rate (FDR).
The FWER measures the probability of making a single false discovery (a Type I error) among the selected features (i.e. selecting one which is not relevant), whereas the FDR measures the proportion of false discoveries made (i.e. the proportion of selected features which are false).
Controlling FWER, however, leads to reduced power (i.e. selecting fewer relevant variables) since it controls the probability of making any false discovery, whereas FDR tries to control the proportion of false discoveries.
Controlling the FDR is important [5; 6; 3].
Often, data-driven feature selection will be used to select a set of candidate features for further investigation.
When further investigation is expensive (for example when further investigation would involve conducting new experiments and collecting more data), a method that cannot control the FDR may result in a large amount of wasted resources, with no guarantee that anything meaningful will be discovered.
On the other hand, being able to control the FDR at, say, 10% ensures that at most, 10% of the spent resources are wasted, and 90% are in fact spent on discovering positive, useful results.
It should be noted, however, that estimating the FDR of a method empirically is hard in practice, since we do not have access to the ground truth relevance.
As such, a theoretical analysis of the method and its (potential) FDR-controlling properties must be carried out, which does not exist for many existing feature selection methods.
[3] is the seminal paper on the knockoff framework, which is an innovative FDR-controlling feature selection method.
Knockoffs are features that are generated to “look like” the real features but be conditionally independent of the label given the real features.
Feature statistics (such as the coefﬁcients of a LASSO [32]) are compared between the real features and their knockoffs and a selection is made when this difference is sufﬁciently large.
Performing the selection in this way allows for an estimate of the FDR to be obtained and the selection threshold can be adjusted to control the FDR at the selected level.
In the original paper, the relationship between the label and the features is constrained to be of a very speciﬁc form; in [7], they remove this constraint and instead provide a theoretical analysis that shifts the burden of knowledge onto knowing the underlying feature distribution.
Unfortunately, while the theoretical results hold for any feature distribution, they rely on being able to generate valid knockoffs, for which [7] only provide a method for generating knockoffs when the distribution is a (known) multivariate Gaussian distribution.
In this paper, we modify the Generative Adversarial Networks (GAN) [11] framework to address this problem, allowing us to generate knockoffs for any distribution (and without any prior knowledge of it).
GANs have been shown to be a powerful method for learning to generate complex distributions [24; 20; 2].
Our main contribution is in modifying the discriminator used in the GAN framework in such a way that the generator learns to generate knockoffs satisfying the necessary swap condition [7] which requires that when a feature and its knockoff are swapped, the joint distribution remains unchanged.
In addition, we propose a method for maximizing the power of our model using Mutual Information
Neural Estimation (MINE) [4] and investigate a regularization method to improve the stability of training.
Our model consists of four networks: (1) a generator network that takes as input noise and the real features, and outputs a set of candidate knockoff features; (2) a discriminator network taking as input “swapped” feature-knockoff features that attempts to determine which variables have been swapped; (3) a Wasserstein GAN discriminator that we use as a regularization term; and (4) a MINE network that estimates the mutual information between each feature-knockoff pair allowing us to maximize the power of the knockoff procedure.
R ELATED W ORKS
Feature selection is a well-studied problem with a wealth of related works ([12; 31; 17; 22] provide a summary of a lot of existing literature); however, most methods do not attempt to control the FDR.
The most common feature selection method for FDR control is the Benjamini-Hochberg (BHq) procedure and its variants [5; 6], which relies on obtaining valid marginal p-values for each selection.
Knockoffs are an active area of research [9; 18; 10].
The notion of a knockoff was ﬁrst introduced in [3] with the theory there requiring that the relationship between the features and the label be of a speciﬁc form.
In [7], they build on the knockoff framework, removing this requirement but instead shifting the requirement to one of knowing the distribution of the features.
As noted in the introduction, the theory in [7] holds independent of the distribution of the features - relying only on being able to generate valid knockoffs (which exist for any distribution of features).
However, they only propose a method for generating knockoffs when the distribution of features is jointly Gaussian.
While they do propose a method for generating approximate knockoffs in the non-Gaussian setting (by simply approximating the features as Gaussian), the guarantees on FDR control do not hold for their approximate knockoffs.
In [26] and [10], they add to the class of constructible knockoffs, describing methods for constructing knockoffs for Markov Chains, Hidden Markov Models and
Gaussian Mixture Models.
Though once again, knowledge of the full distribution is still necessary for their construction.
In this paper we use a framework motivated by GANs [11] to learn to generate knockoffs without any assumptions on the distribution of the features.
To do this, we modify the discriminator so that rather than trying to determine whether a sample is real or fake, it attempts to identify which components have been “swapped”.
In [38], an unconventional discriminator is used that performs component-wise discrimination for the purpose of imputation.
While the problem addressed in that paper is different to the one here, the key idea relies on a similar modiﬁcation to the discriminator to be able to appropriately guide the generator.
In order to maximize the power of our variable selection mechanism, it will be desirable that the feature-knockoff pairs are ”as independent as possible” (this is discussed in [7]).
In order to achieve this we will investigate the use of a promising recent paper, MINE [4].
MINE proposes a neural architecture and training procedure capable of estimating the mutual information between two random variables.
As the mutual information between two random variables is zero only when they are independent, we will use this as a measure of independence and attempt to minimize it during the training of our modiﬁed GAN.
B ACKGROUND
In this section we introduce our notation and deﬁne knockoffs as in [7].
Let us denote the feature space by X and the label space by Y .
Let the dimension of X be d .
Suppose that X =
( X 1 , ..., X d ) and Y are random variables over X and Y .
As in [7], we will work with the notion of a null set .
Deﬁnition 1.
A variable X j is said to be ”null” if and only if Y is independent of X j conditional on { X i : i (cid:54) =
j } .
We deﬁne H 0 to be the set of all null variables.
Our goal will be to discover as many relevant features as possible while controlling the FDR.
For a given (potentially random) selection procedure that selects ˆ S ⊂ { 1 , ..., d } , we deﬁne the FDR to be
[Equation]
Note that this agrees with the usual notion of FDR (i.e. when deﬁned in terms of the Markov blanket) under mild assumptions (for a more thorough discussion see [7]).
3. K NOCKOFFS
Deﬁnition 2.
A knockoff [7] for X is a random variable ˜
X ∈ X satisfying the following two properties:
[Equation]
for all S ⊂ { 1 , ..., d } where ( · , · ) swap(S) denotes the vector obtained by swapping the i th component with the ( i + d ) th component for each i ∈ S and d. = is equality in distribution.
In order to use knockoffs for feature selection, we must deﬁne an appropriate feature statistic, W j , that depends on X , ˜ X and Y , i.e. W j = w j (( X , ˜ X ) , Y ) for some function w j .
This function w j must satisfy the following ﬂip-sign property:
[Equation]
One of the procedures used in [7] to construct these statistics is to perform LASSO, treating the augmented feature-knockoffs as the features on which to regress.
This gives LASSO coefﬁcients b 1 , ..., b 2 d , and the statistic W j is set to be the LASSO Coefﬁcient Difference given by
[Equation]
Note that the FDR control guarantees hold independently of the choice of statistic, but a poorly chosen statistic can signiﬁcantly impact the power of the test.
In particular, using the LASSO Coefﬁcient Difference in non-linear settings can yield few discoveries.
The focus of this paper, however, is on generating the knockoffs, not on the statistic used on top of the generated knockoffs and so in our synthetic experiments, we use a linear model for Y to be able to draw fair comparisons between our model and [7].
In the real world data experiment, we use a statistic based on Random Forests for both methods [37].
The following result from [7] depends only on having obtained knockoffs that satisfy Deﬁnitionand feature statistics satisfying (3) (and in particular do not depend speciﬁcally on using LASSO to obtain the statistics).
Theorem 1.
Let q ∈
Given test statistics, W 1 , ..., W d , satisfying (3), let τ = min (cid:26) t > 0 : 1 + |{ j : W j ≤ − t }| |{ j : W j ≥ t }| ≤ q (cid:27) .
Then the procedure selecting the variables
[Equation]
controls the FDR at level q , i.e.
[Equation]
≤ q.
K NOCKOFF GAN
It should be noted that in order to satisfy equation (2), it simply needs to be the case that the knockoffs are constructed without looking at the label, Y .
In order to satisfy equation (1) we use a modiﬁed
GAN framework, which gives us the ﬂexibility to learn to generate knockoffs without any assumptions on the distribution of the original features.
[Figure]
4. G ENERATOR
The generator, G , will be a function G ( · , · ; φ ) :
X × [0 , 1] c → X , parametrized by φ that takes a realization x of X and random noise, z ∼ U ([0 , 1] c ) , as inputs and outputs knockoff features ˜ x .
We deﬁne ˜
X :
= G ( X , z ) .
We model G as a fully connected neural network with weights φ .
4. D ISCRIMINATOR
The main innovation of our paper is in deﬁning the discriminator.
Equation (1) imposes a condition on the joint distribution of ( X , ˜ X ) and as such we must deﬁne a discriminator with a loss that is (not necessarily uniquely) minimized only for joint distributions satisfying this condition.
To that end, the discriminator, D , will be a function D ( · ; ψ ) :
X × X → [0 , 1] d that takes as input a swapped sample-knockoff pair ( x , ˜ x ) swap ( S ) and outputs a vector in [0 , 1] d with the i th component of D (( x , ˜ x ) swap ( S ) )
corresponding to the probability that i ∈ S .
The discriminator is attempting to detect which variables have been swapped and, intuitively, when the discriminator is unable to determine this, the swapped and unswapped joint distributions must be the same.
The loss we use to train the discriminator is the multi-output cross-entropy loss given by
L D = (cid:88)
[Equation]
where · is the standard dot, 1 = (1 , ..., 1) , S
=
( S 1 , ..., S d ) with S
i =
I ( i ∈ S )
( I is the indicator function) and log is taken element-wise.
The following theorem is our main theoretical result, which states that the training regime employed by KnockoffGAN will result in a procedure that generates valid knockoffs.
Theorem 2.
Equation (4) is maximized (with respect to G )
if and only if equation (1) is satisﬁed by
G .
Proof.
The proof, alongside supporting theoretical results, can be found in the Appendix.
In practice, the sum is too computationally expensive ( O (2 d ) ) to calculate and so we perform stochastic gradient descent using minibatches with S sampled uniformly from { 0 , 1 } d , independently for each sample in the minibatch.
We also found that training with respect to the full loss resulted in a poor performance, particularly when d is large.
We found that the discriminator struggled to learn anything when asked to ﬁnd the full swap vector, and the poor discriminator resulted in a poorly trained generator.
In order to overcome this, we introduce a hint vector - ﬁrst introduced in [38] - that we use to reveal partial information to the discriminator about the swap vector.
We do this by using the hint to reveal some, but not all, of the components of S to the discriminator.
In doing so, we reduce the burden of the discriminator from needing to determine the entire swap vector to only needing to determine some of the swap vector.
Formally, the hint, H , will be a random variable depending on S , that we pass to the discriminator, alongside ( X , ˜ X ) swap ( S ) .
We use the hint to control the amount of information we pass to D about
S before asking D to predict S .
In practice, our hinting mechanism involves sampling a multivariate
Bernoulli random variable, B from i.i.d.
components, which each take value 1 with probability 0 .
The hint is then constructed by setting H i =
S i if B i = 1 and H i = 0 .
5 if B i = 0 .
The discriminator is therefore being asked only to predict the values of S for which B i = 0 ; the others,
D is able to directly infer from H i .
In order to avoid overﬁtting to the hint, it becomes necessary to remove these terms from our loss.
Our loss now becomes
L D = (cid:88)
[Equation]
where (cid:12) denotes element-wise multiplication and the expectation over B is implicit in the expectation over H .
4. S TABILITY
We found that adding a regularization term in the form of a Wasserstein GAN discriminator (with
Gaussian Process (GP) regularization)
[2], f , aided performance.
We note that when equation (1) holds, we must have that X d. =
˜ X
and so the addition of this regularizing term does not affect the optimal solution to our loss.
We model f as a fully connect neural network with weights ν .
The loss is given by
[Equation]
where (cid:15) ∼ U [0 , 1] , ˆ X = (cid:15) X
+ (1 − (cid:15) )
˜ X and η is a hyper-parameter (set to 10 in practice).
Note that we have rewritten the loss to be the negative of the one given in [2], allowing us to write our overall objective as a minimax problem.
This loss is added to the generator loss as an additional regularization term.
[Equation] OWER
As noted in [7], it is intuitive that in order to maximize the power of the knockoff selection procedure, we wish to make X j and ˜ X j as ”independent” as possible.
Doing so ensures that as little as possible of the dependence between the real feature and the label is present between the knockoff and the label; this allows us to determine whether or not the relationship between the feature and label is only through the feature’s correlation with other features, or is in fact a true signal.
In order to achieve maximal independence, we look to minimize the mutual information between each feature and its knockoff.
Actually computing the true mutual information requires access to both the joint density of the feature-knockoff pairs and to the marginal densities of each feature and knockoff, which we do not have.
Instead, we look to a promising recent work, Mutual Information Neural Estimation (MINE [4]), that provides a framework for estimating the mutual information using neural networks.
To do so, they estimate the mutual information between random variables U and V by performing gradient ascent on the following objective:
[Equation]
where P UV denotes the joint measure of ( U, V ) with P U
= (cid:82) V d P UV and P V =
(cid:82) U d P UV denoting the marginal measures.
( n ) denotes the empircal distribution associated with n i.i.d.
samples.
Using MINE we approximate the mutual information between each pair X j and ˜ X j by using d neural networks 1 , T 1 , ..., T d , each parametrized by θ 1 , ..., θ d , that we refer to collectively as the power network , and will write P to denote the collection of networks T 1 , ..., T d .
The mutual information is added using a trade-off parameter λ to the loss for G .
Formally, deﬁne L P by
[Equation]
where κ is a random permutation of [ n ] 2 and ( i ) denotes the i th sample - noting that dependence on
G is through ˜X .
4. F INAL O BJECTIVE
The resulting minimax game played by G , D , W and P is given by
[Equation]
where λ, µ are hyper-parameters (set to 1 in the experiments section).
We train each of G , D , W and P iteratively.
Pseudo-code of our knockoff construction algorithm can be found in Algorithm 1 and a visual representation of our architecture in Fig.
After generating knockoffs, feature statistics are computed according to some procedure (in the synthetic experiments we use LASSO and in the real data experiment we use a Random Forestbased statistic [37]).
Features are then selected based on these statistics according to Theorem 1.
E XPERIMENTS
In this section we demonstrate the capability of our method to match the results of [7] in settings where their model is correctly speciﬁed (i.e. when the underlying feature distribution is Gaussian) and then go on to show, in settings where the underlying feature distribution is non-Gaussian, that our method is able to outperform their Gaussian approximation.
We compare to two versions of the
BHq method [5; 6] to provide a baseline.
We also perform a qualitative analysis of KnockoffGAN on a real-world dataset.
We compare features found by KnockoffGAN to PubMed literature and show that KnockoffGAN discovers several meaningful features for 2 different disease outcomes.
[-(cid:80) 60 i =
-]
1: Inputs: mini-batch size n mb > 0 , Initialize parameters φ, ψ, ν, θ 1 , ..., θ d 2: while Converge do 3:
Discriminator Update 4:
Sample x 1 , ..., x n mb from D , z 1 , ..., z n mb ∼ P
Z 5:
Sample S 1 , ..., S n mb i.i.d ∼ U ( { 0 , 1 } d ) , b 1 , ..., b n mb ∼ Ber (0 . 9) 6: for i = 1 , ..., n mb do 7:
˜ x i ← G ( x i , z i ; φ ) 8: h i =
S i (cid:12) b i + 0 .
5( 1 − b i ) 9:
Update D by ascending its stochastic gradient ∇ ψ n mb (cid:88) i
= (cid:104) ( S i (cid:12) ( 1 − b i )) ·
log( D (( x i , ˜ x i ) swap ( S ) ) , h i )
+ (( 1 − S i ) (cid:12) ( 1 − b i )) · log( 1 − D (( x i , ˜ x i ) swap ( S ) , h i )) (cid:105) 10:
MINE Update 11:
Sample x 1 , ..., x n mb from D , z 1 , ..., z n mb ∼ P Z , κ ∼ U ( S n mb ) 12: for i = 1 , ..., n mb do 13:
˜ x i ← G ( x i , z i ; φ ) 14: for j = 1 , ..., d do 15:
Update T j by ascending its stochastic gradient
[Table]
[Table]
[Table]
x ( κ ( i )) j )) (cid:17) 16:
WGAN-GP Update 17:
Sample x 1 , ..., x n mb from D , z 1 , ..., z n mb ∼ P Z 18: for i = 1 , ..., n mb do 19:
Sample (cid:15) ∼ U [0 , 1] 20: ˜ x i ← G ( x i , z i ; φ ) 21:
ˆ x i = (cid:15) x i + (1 − (cid:15) )
˜ x
i 22:
Update f by ascending its stochastic gradient ∇ ν (cid:80) n mb i =1 (cid:104) f
( x i ) − f (˜ x i ) − η ( ||∇ ˆ x i f
(ˆ x i ) ||
2 − 1) 2 (cid:105) 23:
Generator Update 24:
Sample x 1 , ..., x n mb from D , z 1 , ..., z n mb ∼ P
Z 25:
Sample S 1 , ..., S n mb i.i.d ∼ U ( { 0 , 1 } d ) , κ ∼ U ( S n mb ) 26: for i = 1 , ..., n mb do 27:
˜ x i ← G ( x i , z i ; φ ) 28:
Update G by descending its stochastic gradient ∇ φ ( L D + λ L P + µ L f ) S YNTHETIC D ATA E XPERIMENTS 5.1. S IMULATION SETTINGS Evaluating feature selection methods on real data is difﬁcult as we do not have access to the ground truth. To evaluate KnockoffGAN, we conduct a series of experiments using synthetic data , replicating those carried out in [7] and extending them to more general settings.
In each of the following synthetic experiments, we set the feature dimension to be d = 1000 and the number of samples to be n = 3000 .
For each feature distribution we perform two experiments:
Y-Logit: P ( Y = 1 | X ) =
exp( m ( X ))
(1+exp( m ( X )))
where m ( X ) =
{+(cid:80) 60 i =
+}
αδ i
X i with δ i ∈ {− 1 , 1 } sampled uniformly and then ﬁxed for each experiment.
α controls the strength of the inﬂuence that X has on Y , and in the experiments we vary this (as in [7]).
Note that for the auto-regressive settings (found in Section 5.1.2 and the Appendix) the relevant variables are sampled uniformly at random from among the 1000 features (rather than being the ﬁrst 60); in the non-auto-regressive settings this is not necessary.
We report the True Positive Rate (TPR), which is also commonly referred to as the power , deﬁned as
[Equation]
(6) where S ∗ = { 1 , ..., d } \ H 0 is the set of all non-null features.
We also report the FDR to verify that the methods do indeed control it at the speciﬁed level which we set to be 10%.
Note that we are not using FDR as a metric - a lower FDR is not desirable when we set the threshold to 10%.
In fact, we want the methods to be as close to 10% as possible (so that they are achieving maximum power).
We perform 100 replications of each experiment and report the average TPR and FDR.
5.1. G AUSSIAN SETTINGS
We begin by replicating the setup from [7] in which the underlying feature distribution is Gaussian.
In this setting, we do not expect KnockoffGAN to perform better than the original knockoff framework as the original framework assumes a Gaussian distribution.
Our goal here is simply to achieve a similar performance, demonstrating that little performance is lost even when the distribution is known to be Gaussian.
In the ﬁrst experiment that we replicate from [7], the features are set to be auto-regressive (AR(1)) with Gaussian marginal distributions, i.e. X i =
φX i −
+ Z i with Z i being chosen such that X
i i.i.d.
∼
N (0 , 1 n ) .
In this experiment we vary φ , which determines the correlation between features, rather than α .
We ﬁx α = 3 .
5 for Y-Gaussian and α = 10 for Y-Logit.
The results are reported in Fig.
[Figure]
As in [7], we observe that BHq Marginal, which tests for marginal independence of the feature from
Y , suffers from severely increased FDR as we increase the correlation, invalidating the seemingly good TPR.
To make the remaining results clearer, we omit BHq marginal from the rest of this section.
Aside from this, we see in Fig.
2, that the other methods control the FDR at or very close to the speciﬁed 10% threshold.
We also see that across the entire range of α , KnockoffGAN achieves a very similar TPR to the original Knockoff framework.
In the second experiment, we set the underlying feature distribution to be i.i.d.
Gaussian.
We found in this case also that KnockoffGAN was able to control the FDR and achieve a similar TPR to the original knockoff framework.
More details of this experiment and the results for it can be found in the Appendix
5.1. N ON -G AUSSIAN SETTINGS
We now move on to the key results for the paper in which the underlying feature distribution is no longer Gaussian.
In this setting, we expect to outperform the original Knockoff framework due to the fact that they approximate the distribution as Gaussian.
In particular, when this approximation is poor, the knockoffs are no longer valid and as such no FDR guarantees can be given.
On the other hand, KnockoffGAN does not place any requirements on the distribution of the features and as such is able to generate valid knockoffs.
We performed experiments for several different underlying feature distributions, and found that
KnockoffGAN achieved a higher TPR than the original knockoff framework in all cases, while controlling the FDR at the speciﬁed level.
We give the results for X coming from a 4-Gaussian mixture model in Fig.
3 - results for Uniform, Dirichlet, and other (2 and 3) Gaussian mixture models can be found in the Appendix.
To create our 4-mixture model, we set the means ( m 1 , m 2 , m 3 , m 4 ) of the 4 Gaussians to be:
• m 1 i = 1 for i = 1 to 100 and 0 for i = 101 to 1000 , • m 2 i = 1 for i = 1 to 50 and -1 for i = 51 to 100 and 0 for i = 101 to 1000 , • m 3 i = − 1 for i = 1 to 50 and 1 for i = 51 to 100 and 0 for i = 101 to 1000 , • m 4 i = − 1 for i = 1 to 100 and 0 for i = 101 to 1000 .
We scale the variance of each Gaussian to be such that the overall variance of each feature is 1 n .
[Figure]
T P R
Y-Gauss ,2.
F DR
We see in Fig.
3 that KnockoffGAN consistently outperforms the original knockoff framework, achieving a higher TPR across the entire range of α while consistently controlling the FDR at 10%.
In fact, in the Y -Gaussian setting we see that the original knockoff framework performs almost identically to BHq Maximum Likelihood.
5.1. I MPACT OF WGAN REGULARIZATION
We conclude the synthetic experiments by demonstrating the effect of the WGAN regularizer 3 .
We conduct this experiment using an auto-regressive model with U ( − (cid:112) 3 /n, (cid:112) 3 /n ) marginal distributions.
We ﬁx α = 5 for Y-Logit and α = 2 .
5 for Y-Gauss.
[Figure]
As we see in Fig.
4, the WGAN regularizer has a signiﬁcant effect on the results, with the improvement in some places being almost as much as KnockoffGAN without WGAN makes over the original knockoff framework.
As noted in Section 4.3, there is no trade-off introduced by the inclusion of this regularizer; the optimal solution to the loss is unchanged and therefore this regularization is ”free” in terms of FDR control, but as demonstrated improves TPR performance.
5. R EAL D ATA E XPERIMENT
In this section we use a biobank dataset (with 387 dimensions) to qualitatively analyze the performance of KnockoffGAN.
We use KnockoffGAN to select features for two different outcomes: (1)
Cardiovascular Disease (CVD) and (2) Diabetes and then use PubMed literature to asses the validity of the selected features.
We found that the original knockoff framework was unable to select even the most well-known features (such as Age and Sex for CVD [15]), even when the FDR threshold was increased to 20%.
Therefore, there are no relevant features to report for the original knockoff framework and so Table 1 contains only the features selected by KnockoffGAN that were deemed relevant by PubMed literature.
For this the FDR threshold was set to 5% so that the number of discoveries was manageable for cross-reference with PubMed
[Table]
As we see in Table [Table] KnockoffGAN discovers 9 relevant features for CVD and 8 relevant features for diabetes.
Some of the relevant features, such as Age, Sex and Long-standing illness for CVD are well-known relevant features.
The remaining features are all supported by the literature in PubMed.
While this is a qualitative result (it relies on using PubMed as the ground truth), we do believe this demonstrates that KnockoffGAN is a signiﬁcant improvement over the original knockoff generation procedure.
C ONCLUSION
In this paper we built on the knockoff framework introduced in [3] by developing a novel GAN framework, KnockoffGAN, capable of generating knockoffs with no assumptions on the underlying data.
We demonstrated through a series of experiments on a range of synthetic datasets and on a real world dataset that our method improves on the performance of the original knockoff framework.
While we feel this is a signiﬁcant step towards being able to generate knockoffs for any data, there is still more work to be done.
In particular, generalizing this method to time-series data would be non-trivial, and would be an interesting avenue for further investigation.
A CKNOWLEDGEMENT
The authors would like to thank the reviewers for their helpful comments.
The research presented in this paper was supported by the Ofﬁce of Naval Research (ONR) and the NSF (Grant number:
ECCS1462245, ECCS1533983, and ECCS1407712).
A PPENDIX T HEORETICAL R ESULTS
In order to prove Theorem 2, we use similar techniques to those used in the original GAN paper [11].
In what follows, we analyze the minimax game deﬁned by:
[Equation]
where we have used the version of L D given by equation (4) in the main manuscript (i.e. without hinting)
The theoretical results that follow are proven only for this version of the loss, though we do believe that the theorem holds more generally for the hinting version of the loss - this is backed up by our empirical results demonstrating strict FDR control while using the hint mechanism.
After proving that the optimal solution to this game does indeed provide us with valid knockoffs, we will then show that the additional term L f does not change the optimal solution.
Let p be the density of ( X , ˜ X ) .
We begin by stating a lemma, that follows from a similar proof to Proposition 1 in [11].
Lemma 1.
Let ( x , ˜ x ) ∈
X × X .
Then for a ﬁxed generator, G , the i th component of the optimal discriminator, D ∗ (( x , ˜ x )) is given by
[Equation]
for each i ∈ { 1 , ..., d } .
Proof.
The proof of this involves some basic integral manipulation to get that the objective in (7) can be rewritten as
[Equation]
We then observe that y (cid:55)→ a log y + b log(1 − y ) achieves its maximum in [0 , 1] at aa + b and so the objective is maximized (with respect to D , for ﬁxed G )
when
[Equation]
for each i ∈ { 1 , ..., d } .
With this lemma, we are now able to prove our key result.
Theorem 3.
Equation (7) is maximized (with respect to G )
if and only if equation (1) (in the main paper) is satisﬁed by G .
Proof.
We begin by rewriting our loss, substituting in D ∗ , to give us the following loss for G :
[Equation]
where we note that
[Equation]
Then by inspecting each term in the sum, we see that each term is a KL-divergence term that is minimized only when
[Equation]
and
[Equation]
for every i ∈ { 1 , ..., d } , every S ⊂ { 1 , ..., d } and each ( x , ˜ x ) ∈ X × X .
By iteratively applying equation (9), we get that
[Equation]
proving the theorem.
Lemma 2.
The addition of the term L f to our loss, does not affect the optimal solution to it.
Proof.
By theorem 2, it sufﬁces to show that any distribution satisfying equation (1) also minimizes max f L f .
But we note that, as shown in [2], max f L f (or rather sup f L f ) is the Wasserstein distance between X and ˜ X , which is 0 (and minimal) when X d. =
˜ X .
It therefore sufﬁces to show that equation (1) implies X d. =
˜ X .
Let S = { 1 , ..., d } .
Then if ( X , ˜ X ) satisfy equation (1), we get that ( X , ˜ X ) d. =
( ˜ X , X ) .
Since the joint distributions are equal, it follows that the marginal distributions are equal and so by projecting onto the ﬁrst d variables, we get that X d. =
˜ X .
MINE
We state the key theory used by MINE to estimate the mutual information.
For full details see the original paper, [4].
The mutual information is deﬁned as
[Equation]
where U and V are random variables over some spaces U and V , respectively with joint measure P
UV and marginal measures P U
= (cid:82) V d P UV and P V =
(cid:82) U d P UV , respectively.
The mutual information can also be characterized by the Kullback-Leibler divergence, D KL , as
[Equation]
The Donsker-Varadhan representation then gives us for any two probability measures P and Q over a probability space Ω
[Equation]
where the supremum is taken over all functions T such that the two expectations are ﬁnite.
A simple corollary of this is that ﬁxing a class F of functions (such as a parametrized class { T θ : θ ∈ Θ } ) over which the supremum is taken will provide us with a lower bound for the mutual information that approaches the true mutual information as the class becomes sufﬁciently rich.
MINE [4] ﬁx the class to be parametric in this way - given a ﬁxed neural network architecture, they let F = { T θ : θ ∈ Θ
} be the set of all functions parametrized by this network.
I MPLEMENTATION OF K NOCKOFF GAN
In the experiments, the depth of the generator, discriminator and WGAN-GP networks is set toand power network is set to 3.
The number of hidden nodes in each layer is d/ 4 , d/ 16 d/ 4 for the generator, discriminator, and WGAN-GP, respectively.
For the power network, we use 2 diagonal matrices for each layer to make two hidden nodes for each feature separately.
We use ReLu and tanh as the activation functions for each layer except for the output layer where we use a linear activation function for the generator, power network and WGAN-GP networks and sigmoid activation function for the discriminator network.
The number of samples in each mini-batch is 128.
KnockoffGAN is implemented in tensorﬂow.
D ETAILS OF B ENCHMARKS
We use the following links for the implementations of 3 benchmarks.
• Knockoff: http://web.stanford.edu/group/candes/knockoffs/ software/knockoff/index.html • BHq Max: http://web.stanford.edu/group/candes/knockoffs/ software/knockoff/tutorial-4-r.html • BHq Marginal:
Modifying the original code of BHq Max in http://web.stanford.
edu/group/candes/knockoffs/software/knockoff/tutorial-4-r. html
Except for the knockoff generation step, KnockoffGAN follows the same procedures as the original knockoff framework described at http://web.stanford.edu/group/candes/ knockoffs/software/knockoff/tutorial-2-r.html to select features.
A DDITIONAL E XPERIMENTS I NDEPENDENT G AUSSIANS
In the following experiment, features were taken to be i.i.d.
Gaussian, with mean 0 and variance 1 n , i.e. X ∼ N (0 , 1 n I n ) .
The results are reported in Fig.
[Figure]
[Figure]
F DR
Knockoff GAN Knockoff
BHq Max Lik.
BHq Marginal , 2.
[Figure] [Figure]
Y-Gauss , 2.
F DR
As we see in Fig.
5, all methods control the FDR at or very close to the speciﬁed 10% threshold.
We also see that across the entire range of α , KnockoffGAN achieves a very similar TPR to the original knockoff framework.
N ON -G AUSSIAN SETTINGS I NDEPENDENT U NIFORM
In this experiment we set the feature distribution to be a Uniform distribution with mean 0 and variance 1 n (to be consistent with the Gaussian experiments).
Fig.
6 displays the results for each component of X being i.i.d.
U ( − (cid:112) 3 /n, (cid:112) 3 /n ) .
Once again we see that KnockoffGAN consistently outperforms the original knockoff framework, achieving a higher TPR across the entire range of α in both settings.
[Figure]
Knockoff GAN
Knockoff
BHq Max Lik.
T P R
Y-Gauss ,
[Figure]
D IRICHLET
In this experiment we set the feature distribution to be a Dirichlet (1 , ..., 1) distribution - i.e. the uniform distribution over the ( d − 1) -simplex.
Correlation here exists through the requirement that (cid:80) di =
[Equation]
T P R
[Figure]
F DR
Knockoff GAN
Knockoff
BHq Max Lik.
[Figure]
[Figure]
G AUSSIAN M IXTURE M ODELS
For the GMM2 model we set the means ( m 1 , m 2 ) of the 2 Gaussians to be:
• m 1 i = 1 for i = 1 to 100 and 0 for i = 101 to 1000 , • m 2 i = − 1 for i = 1 to 100 and 0 for i = 101 to 1000 .
T P R
Y-Logit ,
F DR
[Figure]
GAN Knockoff BHq Max Lik.
T
[Figure]
Y-Gauss , 2.
F DR
For the GMM3 model we set the means ( m 1 , m 2 , m 3 ) of the 3 Gaussians to be:
• m 1 i = 1 for i = 1 to 100 and 0 for i = 101 to 1000 , • m 2 i = 1 for i = 1 to 50 and -1 for i = 51 to 100 and 0 for i = 101 to 1000 , • m 3 i = − 1 for i = 1 to 100 and 0 for i = 101 to 1000 . T P R
[Figure]
F DR
[Figure]
BHq
[Figure]
[Figure] [Figure] [Figure]
[Figure]
DR
H YPER - PARAMETER A NALYSIS
Hyper-parameter selection in the feature selection problem is difﬁcult; hyper-parameter selection cannot be performed using cross-validation as we do not have access to ground truth.
The hyperparameters must therefore be ﬁxed a priori.
For this we believe that λ
= 1 and µ = 1 are perhaps the most canonical choice we could make and thus these were used in the main manuscript.
In the following experiments, we investigate the sensitivity of the results to various settings of λ and µ .
The results below are in the auto-regressive Uniform setting with Y-Logit and FDR set to 10%.
[Table] [Table]
As can be seen in Table 2 and 3 , the performance of KnockoffGAN is not sensitive to the value of λ and µ .
The only signiﬁcant difference can be seen when either λ or µ is set to 0 which represents the exclusion of either the power network or WGAN-discriminator network, respectively, from the model.
In particular, the lack of sensitivity to µ aligns with Lemma 2 in which we see that there is not a trade-off between L f and L D but rather the two can be simultaneously minimised.
To further understand the effects of the WGAN network, we report the ﬁnal values of the other losses ( L D and L P ) when µ = 1 and µ =
(i.e. with and without the WGAN regularisation).
The results are given below in the auto-regressive Uniform setting with Y-Logit and FDR set to 10%.
[Table]
As can be seen in Table 4, the inclusion of the WGAN network improves the ﬁnal value of L D , bringing it signiﬁcantly closer to its optimal value of log(2)
≈ 0 .
On the other hand, the power network loss is increased, however, this trade-off is expected and acceptable - the FDR guarantees rely on the discriminator loss ( L D ) and not the power network.
As we see in Table 3, the TPR does not suffer from this increased loss for the power network.
For our ﬁnal hyper-parameter evaluation, we investigate the effect of varying the hinting probability, p , which determines the probability with which B i = 1 .
We note that this hyper-parameter is used to trade-off between speed of learning and optimality of the learned solution.
A low probability makes for fast convergence, but suboptimal convergence whereas a high probability makes for slow convergence (but a more optimal solution).
We chose 0 .
9 to balance this, following the implementation of [38].
To demonstrate this trade-off we vary this hyper-parameter (from 0 to 0.9).
The results below are for the auto-regressive Uniform distribution with Y-Logit and FDR set to 10%.
[Table]
R ESULTS WITH FDR THRESHOLD 5%
In all of the synthetic experiments above, the FDR threshold is set to 10% (which is the level most thoroughly investigated by [3]).
In this ﬁnal experiment, we set the FDR threshold to 5% (which aligns with our real-world experiment) and verify that KnockoffGAN is capable of controlling the
FDR at this level and still outperforms the original knockoff framework.
We conduct the experiments on (1) an auto-regressive distribution with Gaussian marginal distributions, (2) an auto-regressive distribution with Uniform marginal distributions, and (3) a 4-mixture Gaussian mixture model. [Figure]
[Figure]
T P R
Y-Logit ?
F DR
Knockoff GAN
Knockoff
BHq Max Lik.
T P R
Y-Gauss ? 0.
F DR
[Figure]
[Figure]
Y
[Figure]
Logit ,
[Figure]
Knockoff GAN
Knockoff
BHq Max Lik.
[Figure]
Y-Gauss , 2.
[Figure]
DR
S TATISTICS OF REAL - WORLD BIOBANK DATASET
To preserve anonymity of the authors, the full details of this dataset will be given upon acceptance of the paper.
In Table
[Table]
we provide the basic statistics of the real-world biobank dataset
[Table]
Label distribution
[Table]