Normalizing Flows for Knockoff-free Controlled Feature Selection
Abstract
Controlled feature selection aims to discover the features a response depends onwhile limiting the false discovery rate (FDR) to a predeﬁned level.
Recently, multiple deep-learning-based methods have been proposed to perform controlled featureselection through the Model-X knockoff framework.
We demonstrate, however,that these methods often fail to control the FDR for two reasons.
First, these methods often learn inaccurate models of features.
Second, the “swap” property, whichis required for knockoffs to be valid, is often not well enforced.
We propose a newprocedure called F LOW S ELECT to perform controlled feature selection that doesnot suffer from either of these two problems.
To more accurately model the features, F LOW S ELECT uses normalizing ﬂows, the state-of-the-art method for densityestimation.
Instead of enforcing the “swap” property, F LOW S ELECT uses a novel
MCMC-based procedure to calculate p-values for each feature directly.
Asymptotically, F LOW S ELECT computes valid p-values.
Empirically, F LOW S ELECTconsistently controls the FDR on both synthetic and semi-synthetic benchmarks,whereas competing knockoff-based approaches do not.
F LOW S ELECT also demonstrates greater power on these benchmarks.
Additionally, F LOW S ELECT correctlyinfers the genetic variants associated with speciﬁc soybean traits from GWAS data.
Introduction
Researchers in machine learning have made much progress in developing regression and classiﬁcationmodels that can predict a response based on features.
In many application areas, however, practitionersneed to know which features drive variation in the response and they need to do so in a way thatlimits the number of false discoveries.
For example, in genome-wide association studies (GWAS),scientists must consider hundreds of thousands of genetic markers to identify variants associated witha particular trait or disease.
The cost of false discoveries (i.e., selecting variants that are not associatedwith the disease) is high, as a costly follow-up experiment is often conducted for each selectedvariants.
Another example where controlled feature selection matters is analyzing observational dataabout the effectiveness of educational interventions.
In this case, researchers may want to selectcertain educational programs to implement on a larger scale and require conﬁdence that their selectiondoes not include unacceptably many ineffective programs.
As a result, researchers are interested inmethods that model the dependence structure of the data while providing an upper bound on the falsediscovery rate (FDR).
Model-X knockoffs (Candès et al., 2018) is a popular method for controlled variable selection,offering theoretical guarantees of FDR control and the ﬂexibility to use arbitrary predictive models.
However, even with knowledge of the underlying feature distribution, the Model-X knockoffs methodis not feasible unless the feature distribution is either a ﬁnite mixture of Gaussians (Gimenez et al.
,2019) or has a known Markov structure (Bates et al., 2020).
Hence, a body of research explores theuse of empirical approaches that use deep generative models to estimate the distribution of X and sample knockoff features (Jordon et al., 2019; Liu & Zheng, 2018;
Romano et al., 2020; Sudarshanet al., 2020).
The ability of these methods to control the FDR is contingent on their ability to correctly modelthe distribution of the features.
By itself, learning a sufﬁciently expressive feature model can bechallenging.
However, the knockoff procedure requires learning a knockoff distribution that satisﬁesthe swap property , which is a much stronger requirement.
Formally, let X ∈ R D be a sample fromthe feature distribution and ˜ X ∈ R D be a sample from the knockoff distribution conditioned on X .
The swap property stipulates that the joint distribution ( X, ˜ X ) ∈ R 2 D must be invariant to swappingthe positions of any subset of features S ∈ { 1 , . . .
, D } :
[Equation]
Here, swap ( S ) means exchanging the positions of X j and ˜
X
j for all j ∈ S .
For example, inthe case D = 3 and S = { 1 , 3 } , the joint distribution is ( X, ˜ X ) =
( X 1 , X 2 , X 3 , ˜ X 1 , ˜ X 2 , ˜ X 3 ) ,and the swapped joint distribution is ( X, ˜ X ) swap ( S )
=
( ˜ X 1 , X 2 , ˜ X 3 , X 1 , ˜ X 2 , X 3 ) .
Note that, for
S = { 1 , . . .
, D } , the swap property implies that ˜ X D
= X .
See Candès et al.
for a more detaileddescription of the swap property.
Even if a distribution were found satisfying the swap property, it may not provide enough power tomake discoveries.
For example, both properties are trivially satisﬁed by constructing exact copies ofthe features as knockoffs, but the resulting procedure has no power.
It situations where a valid knockoff distribution is available to sample from, knockoffs are computationally appealing because they require only one sample from a knockoff distribution to assess therelevance of all p features.
However, in situations where the joint density of the features is unknown,we show that empirical approaches to knockoff generation (Jordon et al., 2019; Liu & Zheng, 2018;
Romano et al., 2020; Sudarshan et al., 2020) fail to characterize a valid knockoff distribution andtherefore do not control the FDR.
We further show that even with a known covariate model, it is notstraightforward to construct a valid knockoff distribution unless a speciﬁc model structure is known.
We propose a new feature selection method called F LOW S ELECT (Section 3), which does notsuffer from these problems.
F LOW S ELECT uses normalizing ﬂows to learn the joint density of thecovariates.
Normalizing ﬂows is a state-of-the-art method for density estimation; asymptotically,it can approximate any distribution arbitrarily well (Papamakarios et al., 2021; Kobyzev et al.
,2020; Huang et al., 2018).
Additionally, F LOW S ELECT circumvents the need to sample a knockoffdistribution by instead applying a fast variant of the conditional randomization test (CRT) introducedin Candès et al.
Samples from the complete conditionals are drawn using MCMC, ensuringthey are unbiased with respect to the learned data distribution.
Asymptotically, F LOW S ELECT computes correct p-values to use for feature selection (Section 4).
Our proof assumes the universal approximation property of normalizing ﬂows and the convergenceof MCMC samples to the Markov chain’s stationary distribution.
Under the same assumptionsas the CRT, which includes a multiple-testing correction as in Benjamini & Hochberg (1995), aselection threshold can be picked which controls the FDR at a pre-deﬁned level.
Empirically, onboth synthetic (Gaussian) data and semi-synthetic data (real predictors and a synthetic response),
F LOW S ELECT controls the FDR where other deep-learning-based knockoff methods do not.
Incases where competing methods do control the FDR, F LOW S ELECT shows higher power (Section 5).
Finally, in a challenging real-world problem with soybean genome-wide association study (GWAS)data, F LOW S ELECT successfully harnesses normalizing ﬂows for modeling discrete and sequential
GWAS data, and for selecting genetic variants the traits depend on (Section 5.4).
Background
F LOW S ELECT brings together four existing lines of research, which we brieﬂy introduce below.
Normalizing ﬂows Normalizing ﬂows is a general framework for density estimation of a multidimensional distribution with arbitrary dependencies (Papamakarios et al., 2021).
A normalizingﬂow starts with a simple probability distribution (e.g., Gaussian or uniform), which is called thebase distribution and denoted Z , and transforms samples from this base distribution through a series of invertible and differentiable transformations, denoted G , to deﬁne the joint distribution of
X ∈ R D ∼ P X .
A normalizing ﬂow with enough transformations can approximate any multivariatedensity, subject to regularity conditions detailed by Kobyzev et al.
Compared to other densityestimation methods, normalizing ﬂows are computationally efﬁcient.
Details about the speciﬁcnormalizing ﬂow architecture used in F LOW S ELECT are provided in Appendix A.
Controlled feature selection Consider a response Y which depends on a vector of features X ∈ 92 R D .
Depending on how the features are chosen, it is plausible that only a subset of the featurescontains all relevant information about Y .
Speciﬁcally, conditioned on the relevant features in X , Y is independent of the remaining features in X (i.e. the null features).
The goal of the controlledfeature selection procedure is to maximize the number of relevant features selected while limitingthe number of null features selected to a predeﬁned level.
If we denote the total number of selectedfeatures R , then we can decompose R into V , the number of relevant features selected, and S , thenumber of null features selected.
Conditional randomization test Controlled feature selection can be seen as a multiple hypothesistesting problem where there are p null hypotheses, each of which says that feature X j is conditionallyindependent of the response Y given all the other features X − j .
Explicitly, the test of the followinghypothesis is conducted for each feature
j = { 1 , . . .
, D } :
[Equation]
To test these hypotheses, one can use a conditional randomization test (CRT) (Candès et al., 2018).
For each feature tested in a conditional randomization test, a test statistic T j (e.g., the LASSOcoefﬁcient or another measure of feature importance) is ﬁrst computed on the data.
Then, the nulldistribution of T j is estimated by computing its value ˜ T j based on samples ˜
X j drawn from theconditional distribution of X j given X − j .
Finally, the p-value is calculated based on the empirical
CDF of the null test statistics, and features whose p-values fall below the threshold set by the
Benjamini-Hochberg procedure (Benjamini & Hochberg, 1995) are selected.
Though the CRT isintroduced as a computationally inefﬁcient alternative to knockoffs, the CRT nonetheless has appealbecause it requires only knowledge of the feature distribution, which can be learned empirically bymaximum likelihood.
Holdout randomization test The holdout randomization test (HRT)
(Tansey et al., 2021) is a fastvariant of the CRT; it uses a test statistic that requires ﬁtting the model only once.
Let θ representthe parameters of the chosen model, and let T ( X, Y, θ ) be an importance statistic calculated fromthe model with input data.
For example, T , could be the predictive likelihood P θ ( Y test | X test ) or thepredictive score R 2 .
To use the HRT, ﬁrst ﬁt model parameters ˆ θ based on the training data.
Next, foreach covariate j , calculate the test statistic T ∗ j ← T ( X test , Y test , ˆ θ ) .
Then, generate k null samplesand compute T j,k ← T ( X test ( j ← j k ) , Y test , ˆ θ ) , where X test ( j ← j k ) replaces the j -th covariate with the k -thgenerated null sample.
Finally, calculate the p-value as in the CRT, based on the empirical CDF ofthe null test statistics.
Methodology
F LOW S ELECT implements the CRT for arbitrary feature distributions by using a normalizing ﬂow toﬁt the feature distribution and Markov chain Monte Carlo (MCMC) to sample from each completeconditional distribution.
Performing controlled feature selection with F LOW S ELECT consists of thethree steps below.
Step 1: Model the predictors with a normalizing ﬂow
Starting with the observed samples of the features X 1 , . .
, X N ∼ P X , we ﬁt the parameters of anormalizing ﬂow G θ to maximize the log likelihood of the data with respect to a base distribution p Z :
Input: Feature matrix X ∈ R N × D , observation index i , feature index j , number of samples K , ﬁtted normalizing ﬂow p ˆ θ , MCMC proposal q j
Output: Null features ˜
X
i,j,k for k = 1 , . . .
, K for k = 1 , . . .
K do
Propose: X (cid:63)i,j,k ∼ q j ( ·| ˜ X
i,j,k − 1 , X i, − j ) r i,j,k ← p ˆ θ ( X (cid:63)i,j,k ,X i, − j ) q j ( ˜ X i,j,k − 1 | X (cid:63)i,j,k ,X i, − j ) p ˆ θ ( ˜ X i,j,k − 1 ,X i, − j ) q j ( X (cid:63)i,j,k | ˜
X i,j,k − 1 ,X i, − j )
Sample: U i,j,k ∼ Bernoulli ( r i,j,k ∧ 1) if U i,j,k = 1 then ˜
X i,j,k ← X (cid:63)i,j,k else ˜
X
i,j,k ← ˜ X
i,j,k −end if end for ˆ θ = arg max θ
N (cid:88) i
= log p θ ( X i ) where p θ ( X i )
=
p Z ( G θ ( X )) (cid:12)(cid:12)(cid:12)(cid:12) det (cid:18)
∂G θ ( X ) ∂X (cid:19)(cid:12)(cid:12)(cid:12)(cid:12) .
The resulting density p ˆ θ is a ﬁtted approximation to the true density P X .
The speciﬁc normalizingﬂow architecture we use in our ﬁrst two experiments consists of a single Gaussianization layer (Menget al., 2020) followed by a masked autoregressive ﬂow (MAF) (Papamakarios et al., 2017).
Theﬁrst layer can learn complex marginal distributions for each covariate, while the MAF learns thedependencies between them.
More detail on normalizing ﬂows and on this particular architecture canbe found in Appendix A.
Step 2: Sample from the complete conditionals with MCMC
For each feature j , we aim to sample corresponding null features
˜ X
i,j,k for all k ∈ { 1 , . . .
, K } thatare equal in distribution to p ˆ θ ( X i,j | X i, − j ) , but independent of Y i .
However, directly sampling fromthis conditional distribution is intractable.
Instead, we implement an MCMC algorithm that admitsit as a stationary distribution.
The samples drawn from MCMC are autocorrelated, but any statisticcalculated over these samples will converge almost surely to the correct value.
The choice of the
MCMC proposal distribution q j is ﬂexible.
Because each Markov chain is only one-dimensional, a
Metropolis-Hastings Gaussian random walk with the standard deviation set based on the covariancecan be expected to mix rapidly.
Alternatively, information from p ˆ θ , such as higher-order derivatives,could be used to construct a more efﬁcient proposal.
Algorithm 1 details how to implement step 2.
Step 3: Test for signiﬁcance with the HRT
As in the CRT, feature j has high evidence of being signiﬁcant if, under the assumption that j is a nullfeature, the probability of realizing a test statistic greater than the observed T j ( X ) is low.
Formally,letting [ ˜ X j , X − j ] be the observed feature matrix with the observed feature X j swapped out with thenull feature ˜ X j , we can write this as a p-value α j :
[Equation]
[Equation]
However, the above p-value α j is not tractable.
For each sample ˜ X · ,j,k drawn using MCMC, wecalculate the corresponding feature statistic and compare it to the real feature statistic, leading to an approximated p-value ˆ α j :
[Equation]
To control the FDR, we use the Benjamini-Hochberg procedure to establish a threshold for theobserved p-values.
Speciﬁcally, we set the threshold as s ( γ ) (cid:44) max j { ˆ α j : ˆ α j ≤ jD γ } , and select allfeatures j such that α j ≤ s ( γ ) .
The Benjamini-Hochberg correction only guarantees FDR control provided that the p-values haveeither positive or zero correlation.
Thus, the FDR control of F LOW S ELECT depends on theseassumptions being met.
A more conservative correction from Benjamini & Yekutieli (2001) allowsfor arbitrary dependencies in p-values, but it suffers from low power.
The Benjamini-Hochbergcorrection is widely used and empirically robust (Tansey et al., 2021), so we report results using it.
Across our synthetic and semi-synthetic benchmarks in Section 5, we also ﬁnd that F LOW S ELECTmaintains empirical FDR control.
Provided that the Benjamini-Hochberg assumptions are met, the FDR will be controlled, but thepower of the test depends on T j being higher when j is a signiﬁcant feature.
For example, if Y isexpected to vary approximately linearly with respect to X , T j ( X ) could be the absolute estimatedregression coefﬁcient | ˆ β j | for the linear model Y =
Xβ + (cid:15) .
Another choice is the HRT featurestatistic described earlier.
Asymptotic results
The ability of F LOW S ELECT to control the FDR relies on its ability to produce estimated p-valuesthat converge to the correct p-values for the hypothesis test in Equation (2).
Given asymptoticallycorrect p-values, rejecting according to the threshold set by Benjamini-Hochberg gives FDR control.
Theorem 1.
Let X ∈ R N
×
D be a random feature matrix, where each row X i, · is independent andidentically distributed; x ∈ R N × D be the observed feature matrix; and α j be the p-value as deﬁnedin Equation (4) with test statistic T j ( X ) .
Suppose there exists a sequence of functions ( G n ) ∞ n =1 anda base
random variable
Z satisfying the following conditions:
Each G n is continuously differentiable and invertible.
G n → G pointwise for some map G that is triangular, increasing, continuously differentiable, and satisﬁes G
( X i, · ) D = Z .
For n = 1 , 2 , . . .
, let X n be the random feature matrix where each row i is independent and hasdistribution X ni, · = ( G n ) − 1 ( Z ) .
Then, the p-value in Equation (5) calculated using K MCMCsamples targeting X n · ,j | X n · , − j = x · , − j converges to the correct p-value α j with probability 1 .
Proof.
Here we sketch the proof.
A full proof can be found in Appendix B. First, by constructioneach G n deﬁnes a distribution X ni, · D = ( G n ) − 1 ( Z ) that in turn implies a conditional distribution
X n · ,j | X n · , − j = x · , − j .
We show these conditional distributions converge to the true conditionaldistribution of X · ,j given X · , − j = x · , − j .
Consequently, the probability of observing a higher teststatistic under the approximated null distribution ˜ X n · ,j D = X n · ,j , written α nj , will converge to theprobability under the true null distribution ˜ X · ,j | X · , − j = x · , − j , i.e. α j .
Next, the Cesaro average of
K samples from an MCMC algorithm targeting ˜ X n · ,j | X · , − j = x · , − j , written ˆ α j,K,n will converge to 191 α nj with probability 1 as K → ∞ .
Combining these two convergences leads to the stated result.
The existence of a sequence ( G n ) ∞ n =1 that converges to the true mapping G depends on the family ofnormalizing ﬂows chosen.
Universality has been show for a variety of normalizing ﬂows (Huang et al.
Meng et al., 2020; Kobyzev et al., 2020), including the Gaussianization Flows and Masked
Autoregressive Flows used in our experiments.
In practice, it is unlikely that an exact mapping G willbe learned, as doing so could require inﬁnite training data, inﬁnitely deep transformations, and exactnonconvex optimization.
Nonetheless, normalizing ﬂows work extremely well in practice; Theorem 1gives intuition for the good performance of F LOW S ELECT that we observe empirically.
Experiments
[Figure]
5.1 Synthetic experiment with a mixture of highly correlated Gaussians
We compare F LOW S ELECT to the aforementioned knockoff methods with synthetic data drawn froma mixture of three highly correlated Gaussian distributions with dimension D = 100 .
For eachknockoff method, we use the exact implementation described in their respective papers, and we utilizethe code made publicly available by the authors (c.f. Appendix D.3 for further details).
For furthercomparison, we also implement the MASS knockoff procedure from Gimenez et al.
(2019) and the RANK knockoff procedure from Fan et al. These methods estimate the unknown featuredistribution using either a mixture of Gaussians (MASS) or a sparse precision matrix (RANK), andthen sample the knockoffs directly as in Candès et al.
To generate the data, we draw N = 100 , 000 highly correlated samples.
For i = 1 , . . .
, N , we sample
[Equation]
i
[Equation]
(cid:88)
[Equation]
(6) with mixing weights π = (0 . 371 , 0 .
258 , 0 . 371) , mean vector µ = (0 , 20 , 40) , and covariancematrices Σ m .
Each covariance Σ m follows an AR(1) pattern such that (Σ m ) i,j = ρ | i − j | m whereρ =
j .
The response Y i is linear in f i ( X i ) for some function f
i and coefﬁcientvector β i.e., Y i =
f i ( X i ) β + (cid:15) i .
Each coefﬁcient β
j equals 100 √ N B j , where B j = 0 with probability0 .
8 , B j = 1 with probability 0.1, and B j = − 1 with probability 0.1.
We consider two differentschemes for the f i that connect the features to the response.
In our linear setting, f i is equal to theidentity function.
In our nonlinear setting, f i ( x ) is set equal to sin(5 x ) for odd i and f i ( x ) = cos(5 x )for even i .
The experimental setting we have described so far is adapted from Sudarshan et al.
However,we found that the N = 2000 they used was too few observations for any of the methods to do wellin a general non-linear setting.
Moreover, in many situations where controlled feature selection isdeployed, neighboring features will be highly correlated.
To reﬂect this, we also increased the basecorrelation between features within each mixture to create a more challenging example.
We showresults under the original settings of Sudarshan et al.
(2020) in Appendix K.
For each model, we use 90% of the data for training to generate null features and the remaining 10%for calculating the feature statistics.
To deﬁne the feature statistics, we use the holdout randomizationtest (HRT) described at the end of Section 2. Within the HRT, we employ different with differentpredictive models for each response type (“linear” and “nonlinear”).
Speciﬁcally, for the linearresponse, we use the predictive log-likelihood from theLASSO (Tibshirani, 1996), and for thenonlinear response, we use the predictive negative mean-squared error from the random forest(Breiman, 2001).
First, we look at how each procedure models the covariate distribution in Figure 1.
In order to bevalid knockoffs, the distribution of two knockoff features needs to be equal to that of the covariates.
scRNA
[Figure]
In this challenging example, each of the empirical knockoff methods fails to match the ground truth.
In particular, DDLK and DeepKnockoffs are over-dispersed, while KnockoffGAN suffers from modecollapse.
These ﬁndings for DeepKnockoffs and KnockoffGAN are similar to those reported by
Sudarshan et al.
Other than MASS, which directly ﬁts a mixture of Gaussians, F LOW S ELECTis the only method that matches the basic structure of the ground truth.
Figure 2 shows that the empirical knockoff procedures fail to control the FDR for both linear andnonlinear responses.
One explanation for this lack of FDR control is the inability of the deeplearning-based methods to accurately model a knockoff distribution (c.f., Figure 1).
As a result, theassumptions for the knockoff procedure will not hold, and FDR control is not guaranteed.
The effects of misspeciﬁcation are clearly visible in the case of RANK, which estimates the mixtureof-Gaussians dataset with a multivariate Gaussian. However, even MASS, which correctly speciﬁesthe underlying distribution, does not achieve across-the-board FDR control.
This highlights thepotential sensitivity of knockoffs to parameter misﬁt even when the underlying distributional familyof the features is known.
This is conﬁrmed by the fact that, when provided the true parameters, theoracle Model-X maintains FDR control, though with signiﬁcantly less power than F LOW S ELECT .(c.f.
Appendix H).
5.2 Semi-synthetic experiment with scRNA-seq data
In this experiment, we use single-cell RNA sequencing (scRNA-seq) data from 10x Genomics (10x
Genomics, 2017).
Each variable X n,g is the observed gene expression of gene g in cell n .
Thesedata provide an experimental setting that is both realistic and, because gene expressions are oftenhighly correlated, challenging.
More background information about scRNA-seq data can be found in
Agarwal et al.
We normalize the gene expression measurements to lie in [0 , 1] , and we add a small amount of
Gaussian noise so that the data is not zero-inﬂated.
As in the semi-synthetic experiment from
Sudarshan et al.
(2020), we pick the 100 most correlated genes to provide a challenging, yet realisticexample.
We simulate responses that are both linear and nonlinear in the features.
Figure 2 showsthat F LOW S ELECT maintains FDR control across multiple FDR target levels, feature statistics, andgenerated responses.
In cases where the knockoff methods control FDR successfully, F LOW S ELECThas higher power in discovering the features the response depends on.
An advantage of knockoffs over CRT-based methods like F LOW S ELECT is that the predictive modelonly needs to be evaluated once.
Hence, while F LOW S ELECT has a faster runtime than DDLK for this experiment, it is slower than DeepKnockoff and KnockoffGAN.
However, Figure 2 shows thatthese two models fail to reliably control FDR and have much less power than F LOW S ELECT ; it is notclear how additional computational resources could be leveraged to improve the performance of thesecompeting methods.
A full table of runtimes on the scRNA-seq dataset can be found in Appendix F.
The need to compute a different predictive model for each feature within the CRT is mitigated byusing efﬁcient feature statistics such as the HRT
(Tansey et al., 2021) and the distilled CRT (Liuet al., 2020).
These methods ﬁt a larger predictive model once, then evaluate either the residuals ortest mean-squared-error for each feature individually.
Moreover, the ability to scale to large featuredimensions D is more limited by ﬁtting the feature distribution than computational burden, a traitshared by both knockoff- and CRT-based methods.
F LOW S ELECT provides asymptotic guarantees of FDR control assuming sufﬁcient MCMC sampleshave been drawn for the p-values to converge.
In this experiment, the consequence of terminating
MCMC sampling before convergence is low power, rather than loss of FDR control (see Figure 7 in
Appendix J).
Even for small numbers of MCMC samples, the FDR stabilizes below the target rate,while the power steadily increases with the number of samples.
Because the MCMC run is initializedat the true features, we speculate that the sampled features will be highly correlated with the truefeatures in the beginning of the run, making it harder to reject the null hypothesis that a feature isunimportant.
5.3 Ablation Study
F LOW S ELECT differs from the competing knockoff-based approaches in two ways: using normalizingﬂows with MCMC to model the feature distribution for sampling null features and using the CRTfor feature selection.
To illustrate the impact of each of these components separately, we compare tothe procedure used in Tansey et al.
(2021), which uses mixture density networks (MDNs) to modelthe complete conditional distribution of each feature P ( X j |
X − j ) separately.
They then sample nullfeatures from these learned distribution directly and use the HRT for feature selection.
Since both
F LOW S ELECT and this procedure utilize the HRT, this allows us to evaluate whether the performanceimprovement of F LOW S ELECT over empirical knockoffs is solely due to use of the HRT.
We compare the MDN-based approach to F LOW S ELECT on the mixture-of-Gaussians (Section 5.1)and scRNA-seq (Section 5.2) datasets.
A plot of this comparison can be found in Appendix G.
While the MDN-based approach was able to match the performance of F LOW S ELECT on the scRNAseq dataset, it failed to control FDR at any level on the Mixture-of-Gaussians dataset, indicatingthat MDNs are less ﬂexible than normalizing ﬂows.
In aggregate, these results show that both thenormalizing ﬂows paired with MCMC and the use of the HRT for signiﬁcance testing are key to theperformance of F LOW S ELECT .
5.4 Real data experiment: soybean GWAS
Genome-wide association studies are a way for scientists to identify genetic variants (single-nucleotidepolymorphisms, or SNPs) that are associated with a particular trait (phenotype).
We tested F LOW S ELECT on a dataset from the SoyNAM project (Song et al., 2017), which is used to conduct GWASfor soybeans.
Each feature X j takes on one of four discrete values, indicating whether a particular
SNP is homozygous in the non-reference allele, heterozygous, homozygous in the reference allele, ormissing.
A number of traits are included in the SoyNAM data; we considered oil content (percentagein the seed) as the phenotype of interest in our analysis.
There are 5,128 samples and 4,236 SNPs intotal.
To estimate the joint density of the genotypes, we used a discrete ﬂow (Tran et al., 2019).
Modelingof genomic data is typically done with a hidden Markov model (Xavier et al., 2016); however, sucha model may fail to account for long range dependence between SNPs, which a normalizing ﬂowis better suited to handle.
Having a more ﬂexible model of the genome enables F LOW S ELECT toprovide better FDR control for assessing genotype/phenotype relationships.
For the predictive model,we used a feed-forward neural network with three hidden layers.
Additional details of training andarchitecture are presented in Appendix E.
A graphical representation of our results is shown as a Manhattan plot in Figure 3, which plots thenegative logarithm of the estimated p-values for each SNP.
At a nominal FDR of 20%, we identiﬁed
Chromosome - l og 10 ( p )
seven SNPs that are associated with oil content in soybeans.
We cross-referenced our discoverieswith other publications to identify SNPs that have been previously shown to be associated withoil content in soybeans.
For example, F LOW S ELECT identiﬁes one SNP on the 18th chromosome,
Gm18_1685024 , which is also selected in Liu et al.
F LOW S ELECT also selects a SNP on the5th chromosome, Gm05_37467797 , which is near two SNPs ( Gm05_38473956 and Gm05_38506373 )identiﬁed in Cao et al.
but which are not in the SoyNAM dataset.
Sonah et al.
(2014) identiﬁeseight SNPs near the start of the 14th chromosome,
and we select multiple SNPs in a nearby region onthe 14th chromosome (seen in the peak of dots on chromosome 14 in Figure 3).
However, the datasetin Sonah et al.
(2014) is much larger ( ≈ 47 , 000 SNPs), which prevents an exact comparison.
A listof all SNPs selected by our method is provided in Appendix E.
For this experiment, F LOW S ELECTtests over 4000 features in 10 hours using a single GPU.
None of the empirical knockoff procedures(Sudarshan et al., 2020; Jordon et al., 2019; Romano et al., 2020) tested more than 387 features.
Thisshows the potential for F LOW S ELECT for high-dimensional feature selection with FDR control in areasonable amount of time.
Additional details about this experiment are available in Appendix E.
Discussion
F LOW S ELECT enables scientists and other practitioners to discover features a response depends onwhile controlling false discovery rate, using an arbitrary predictive model; even large-scale nonlinearmachine learning models can be utilized.
By making fewer false discoveries for a ﬁxed sensitivitylevel, F LOW S ELECT can reduce the cost of follow-up experiments by limiting the number of irrelevantfeatures considered.
In contrast to the original model-X knockoffs method, F LOW S ELECT does notrequire the feature distribution to be known a priori, nor does it require the feature distribution tohave a particular form (e.g., Gaussian).
Neither of these conditions are often satisﬁed in practice.
A limitation shared by both the conditional randomization test (CRT) and knockoffs is low powerin cases where important features are highly correlated with other important features.
To mitigatethis, one can modify the CRT to test the signiﬁcance of groups of correlated features.
Within the
F LOW S ELECT framework, this entails modifying the MCMC step to draw null samples of groups offeatures conditioned on the others. The group’s p-value can then be calculated with the same holdoutrandomization test (HRT) statistic used for testing individual features. Group feature selection hasalso been explored in knockoffs (Dai & Barber, 2016; Liu et al., 2020). F LOW S ELECT uses normalizing ﬂows which are the state-of-the-art for density estimation, and whichcontinue to be developed.
However, the ﬂexibility of normalizing ﬂows comes at a cost.
Sufﬁcienttraining examples are needed to learn the feature distribution, limiting applicability in data-starvedregimes.
As we show in Appendix K, F LOW S ELECT fares no worse than competing methods inlow-data settings.
In these regimes, F LOW S ELECT could also use other density estimation techniquessuch as autoregressive models.
However, learning the feature distribution is not the sole problem that the competing deep-learningbased knockoff methods face.
To demonstrate this, for DDLK, which also ﬁts a joint distributionas part of its training procedure, we substituted the the exact joint density via an oracle, yet neitherthe empirical FDR nor the power improved signiﬁcantly (c.f. Appendix I).
This points to failure toenforce the swap property, a challenging task given that the number of swaps grows exponentially withthe number of features.
This curse of dimensionality is unavoidable, since the knockoff distributionmust be invariant to all possible swaps for theoretical guarantees to hold.
Unfortunately, not meetingthese guarantees defeats the purpose of using knockoffs in the ﬁrst place.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Runtime discussion in section 5.
and limitations of normalizing ﬂows in section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Theorem 1statement lists all assumptions.
(b) Did you include complete proofs of all theoretical results?
[Yes] Yes; Appendix Bcontains the proof to Theorem 1.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code includedas supplement, and speciﬁc details were described in the Appendix.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] In Appendices D and E.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] All ﬁgures comparing FDR and Power results (Figure2 in main text, Figures 4-6, 8 in the Appendix) contain error bars indicating resultsacross runs.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] We specify hardware used inAppendix F.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix C
(b) Did you mention the license of the assets?
[Yes] See Appendix C
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A] Data are publicly available(e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]